# Apache Spark

Apache Spark is a unified engine for large-scale data analytics.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Introducing Amazon EMR Serverless Flex: Run non-urgent Spark jobs at a discount

DevFeed: [Introducing Amazon EMR Serverless Flex: Run non-urgent Spark jobs at a discount](<https://devfeed.tech/articles/introducing-amazon-emr-serverless-flex-run-non-urgent-spark-jobs-at-a-discount-144577.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/introducing-amazon-emr-serverless-flex-run-non-urgent-spark-jobs-at-a-discount/>)

Author: Neil Mukerje

Published: 2026-11-19T20:58:07Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>)

Tags: [amazon-emr](<https://devfeed.tech/tags/amazon-emr.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [batch](<https://devfeed.tech/tags/batch.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [billing](<https://devfeed.tech/tags/billing.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [discount](<https://devfeed.tech/tags/discount.md>), [driver](<https://devfeed.tech/tags/driver.md>), [flex](<https://devfeed.tech/tags/flex.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [retries](<https://devfeed.tech/tags/retries.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [spark](<https://devfeed.tech/tags/spark.md>), [standard](<https://devfeed.tech/tags/standard.md>)

### AI overview

Amazon EMR Serverless Flex runs non-urgent Spark batch jobs on spare capacity at a fixed 60% discount on worker vCPU and memory charges compared with Standard. It automates Availability Zone and instance selection, protects the driver on a Standard worker, and retries tasks when Flex workers are reclaimed; optional Serverless Storage can reduce recomputation for shuffle-heavy jobs. Flex may start more slowly and run longer than Standard, and a job fails if no executor joins the driver within 20 minutes. In the described 103-query benchmark, all queries completed without resubmission; runtime was 7% longer at 1 TB and 30% longer at 3 TB, while estimated cost fell about 63% at both scales. Results vary with workload and spare capacity. Flex is enabled per job, so teams can compare it with Standard on flexible-deadline workloads.

### Source excerpt

Amazon EMR Serverless Flex is a new execution option that runs non-urgent Spark batch jobs on spare compute capacity at a fixed 60% discount compared to standard pricing. Flex handles Availability Zone selection, instance diversification, driver protection, and automatic retries, so jobs complete even when spare capacity is reclaimed.

## Aderant modernizes analytics data pipeline with Apache Iceberg and AWS Lambda

DevFeed: [Aderant modernizes analytics data pipeline with Apache Iceberg and AWS Lambda](<https://devfeed.tech/articles/aderant-modernizes-analytics-data-pipeline-with-apache-iceberg-and-aws-lambda-141916.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/aderant-modernizes-analytics-data-pipeline-with-apache-iceberg-and-aws-lambda/>)

Author: Sham Birbahadur

Published: 2026-11-19T00:36:23Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [event-driven data processing](<https://devfeed.tech/topics/event-driven-data-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [AWS Database Migration Service](<https://devfeed.tech/topics/aws-database-migration-service.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [acid](<https://devfeed.tech/tags/acid.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [batch](<https://devfeed.tech/tags/batch.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cost](<https://devfeed.tech/tags/cost.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [faster](<https://devfeed.tech/tags/faster.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [legal](<https://devfeed.tech/tags/legal.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [queries](<https://devfeed.tech/tags/queries.md>)

### AI overview

Aderant replaced Stridyn's 15-minute batch pipeline with an event-driven medallion architecture that processes CDC changes from 700 client environments in single-digit seconds to bronze-layer Apache Iceberg tables. AWS DMS delivers Parquet files to S3; S3 notifications and SQS invoke Lambda for ingestion, while scheduled AWS Glue jobs transform data through silver and gold layers. Incremental Glue jobs handle 77 standard gold tables; 34 complex tables use Iceberg materialized views and snapshot comparisons to send changed rows to native Amazon Redshift tables. Aderant reports 10-26x faster queries on native Redshift tables, under-100-millisecond dashboard queries, and 77 percent lower pipeline compute costs versus streaming alternatives. The account says this approach supports schema changes and concurrent tenant workloads; it also describes a possible future Kinesis path for high-volume tenants.

### Source excerpt

Learn how Aderant modernized the data pipeline behind its Stridyn legal analytics platform with Apache Iceberg, AWS Lambda, and AWS Glue, moving from 15-minute batch cycles to an event-driven medallion architecture that processes change data capture events in single-digit seconds for 700 tenants, with 10-26x faster queries and 77 percent lower cost.

## Two decades of The Apache Way: Celebrating 20 years of Google and the Apache Software Foundation

DevFeed: [Two decades of The Apache Way: Celebrating 20 years of Google and the Apache Software Foundation](<https://devfeed.tech/articles/two-decades-of-the-apache-way-celebrating-20-years-of-google-and-the-apache-software-foundation-144106.md>)

Original publisher: [Read original article](<http://opensource.googleblog.com/2026/11/two-decades-of-the-apache-way-google-asf-20-years.html>)

Author: Google Open Source (noreply@blogger.com)

Published: 2026-11-18T19:30:00Z

Content type: article

Language: en

Sources: [Google Open Source Blog](<https://devfeed.tech/sources/google-open-source-blog.md>)

Topics: [Open Source](<https://devfeed.tech/topics/open-source.md>), [open source maintenance](<https://devfeed.tech/topics/open-source-maintenance.md>), [engineering-culture](<https://devfeed.tech/topics/engineering-culture.md>), [Cloud Dataflow](<https://devfeed.tech/topics/cloud-dataflow.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [gemma](<https://devfeed.tech/topics/gemma.md>)

Tags: [2006](<https://devfeed.tech/tags/2006.md>), [2016](<https://devfeed.tech/tags/2016.md>), [ai](<https://devfeed.tech/tags/ai.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-2-0](<https://devfeed.tech/tags/apache-2-0.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [apache-software-foundation](<https://devfeed.tech/tags/apache-software-foundation.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [asf](<https://devfeed.tech/tags/asf.md>), [batch](<https://devfeed.tech/tags/batch.md>), [beam](<https://devfeed.tech/tags/beam.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-dataflow](<https://devfeed.tech/tags/cloud-dataflow.md>), [code](<https://devfeed.tech/tags/code.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [communities](<https://devfeed.tech/tags/communities.md>), [community](<https://devfeed.tech/tags/community.md>), [google](<https://devfeed.tech/tags/google.md>), [google-supports-oss](<https://devfeed.tech/tags/google-supports-oss.md>), [open-source](<https://devfeed.tech/tags/open-source.md>)

### AI overview

Google's relationship with the Apache Software Foundation (ASF), formalized in 2006, continues through sponsorship, upstream contributions, and shared support for open, vendor-neutral, community-led development. Google says its contributor license agreements draw on ASF models and Apache 2.0 is its preferred code license, also used for open hardware and models such as Gemma 4. Examples of collaboration include donating Dataflow SDK components that helped establish Apache Beam; managed Google Cloud services built on Airflow and Spark; and BigQuery support for Apache Iceberg. Google was the ASF's first Platinum Sponsor and contributes funding and cloud infrastructure; Googlers also serve as committers and PMC members, while ASF has participated in Google Summer of Code for over 20 years. The text points to open data, cloud infrastructure, and AI as areas for future collaboration.

### Source excerpt

by Mike Bufano, Open Source Programs Office In 2006, Google formalized its relationship with the Apache Software Foundation (ASF) as one of its first corporate sponsors and twenty years later, that partnership continues through ongoing sponsorship and upstream contributions. At the core of our collaboration is a shared belief in "The Apache Way" and its principles of open, vendor-neutral, and community-led development. From the early days of corporate open source to modern machine learning and open data systems, Google and ASF communities have collaborated on widely used open source software. Shaping how Google does open source When we established Google's open source contribution workflow, we drew on established ASF practices. Both our Individual and Corporate Contributor License Agreements (CLAs), which external developers sign before contributing to a Google-owned open source project, are based on the ASF's CLAs. The Apache 2.0 license also remains Google's preferred open source license for code. Alongside our traditional software releases, we also use Apache 2.0 for open hardware designs and open models like Gemma 4. Co-engineering upstream and in the cloud Across Google Cloud, our engineering teams contribute code upstream alongside Apache project communities and build managed services directly around those open standards: Google Cloud Dataflow & Apache Beam: Drawing on Google's internal Flume and Dataflow systems, Google donated the original Dataflow SDK and runners to the Apache Incubator in 2016 to help establish Apache Beam as a vendor-neutral open standard for unified batch and stream processing. Google Cloud Managed Service for Apache Airflow & Apache Airflow: Managed workflow orchestration built on Apache Airflow, helping engineering and data teams author, schedule, and monitor complex pipelines. Google Cloud Managed Service for Apache Spark & Apache Spark: Managed big data and analytics processing built on Apache Spark, running directly on Google's clou

## LinkedIn Cuts Daily Spark Shuffle Fetch Failures from 750 to 20 with Apache Celeborn

DevFeed: [LinkedIn Cuts Daily Spark Shuffle Fetch Failures from 750 to 20 with Apache Celeborn](<https://devfeed.tech/articles/linkedin-cuts-daily-spark-shuffle-fetch-failures-from-750-to-20-with-apache-celeborn-140877.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/11/linkedin-celeborn-remote-shuffle/>)

Author: Leela Kumili

Published: 2026-11-18T15:05:00Z

Content type: news

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [control-plane](<https://devfeed.tech/topics/control-plane.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [application](<https://devfeed.tech/tags/application.md>), [architecture-design](<https://devfeed.tech/tags/architecture-design.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [big-data-infrastructure](<https://devfeed.tech/tags/big-data-infrastructure.md>), [cloud-architecture](<https://devfeed.tech/tags/cloud-architecture.md>), [compression](<https://devfeed.tech/tags/compression.md>), [compute](<https://devfeed.tech/tags/compute.md>), [compute-costs](<https://devfeed.tech/tags/compute-costs.md>), [control-plane](<https://devfeed.tech/tags/control-plane.md>), [cost](<https://devfeed.tech/tags/cost.md>), [costs](<https://devfeed.tech/tags/costs.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [development](<https://devfeed.tech/tags/development.md>), [devops](<https://devfeed.tech/tags/devops.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [event-driven-architecture](<https://devfeed.tech/tags/event-driven-architecture.md>), [fault-tolerance](<https://devfeed.tech/tags/fault-tolerance.md>), [flink](<https://devfeed.tech/tags/flink.md>), [linkedin](<https://devfeed.tech/tags/linkedin.md>), [linkedin-celeborn-remote-shuffle](<https://devfeed.tech/tags/linkedin-celeborn-remote-shuffle.md>), [news](<https://devfeed.tech/tags/news.md>), [optimizations](<https://devfeed.tech/tags/optimizations.md>), [performance-tuning](<https://devfeed.tech/tags/performance-tuning.md>), [platform](<https://devfeed.tech/tags/platform.md>), [remote](<https://devfeed.tech/tags/remote.md>), [replication](<https://devfeed.tech/tags/replication.md>), [runtimes](<https://devfeed.tech/tags/runtimes.md>), [spark](<https://devfeed.tech/tags/spark.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

LinkedIn deployed Apache Celeborn as a shared remote shuffle service for most Spark and Flink workloads, separating shuffle storage from compute. Map tasks push intermediate data to dedicated workers, and reducers fetch it directly; this reduces contention and dependence on executor lifecycles. Compared with its previous Magnet implementation, LinkedIn reports daily shuffle fetch failures falling from about 750 to 20, compute costs down 10%, and application runtimes improved 14%. Reliability work included a five-node Raft control plane across availability zones, network-aware replication, and health-based worker exclusion; task failures during infrastructure disruptions fell 20-fold. LinkedIn also reports 40% shorter shuffle duration after moving storage to NVMe, a 35% smaller shuffle-worker fleet with Zstandard instead of LZ4, and nearly 90% lower shuffle cost per task. The deployment required substantial engineering beyond the upstream project.

### Source excerpt

LinkedIn deployed Apache Celeborn as a shared remote shuffle platform across most Spark and Flink workloads, reducing daily shuffle fetch failures from approximately 750 to 20. The deployment also lowered compute costs by 10%, improved application runtimes by 14%, and reduced shuffle costs per task by nearly 90% through dedicated storage, replication, and data-plane optimizations. By Leela Kumili

## Getting started with Apache Polaris Catalog in Fivetran's Managed Data Lake Service | Blog | Fivetran

DevFeed: [Getting started with Apache Polaris Catalog in Fivetran's Managed Data Lake Service | Blog | Fivetran](<https://devfeed.tech/articles/getting-started-with-apache-polaris-catalog-in-fivetran-s-managed-data-lake-service-blog-fivetran-140235.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/getting-started-with-apache-polaris-catalog-in-fivetrans-managed-data-lake-service>)

Author: Andrew Madson

Published: 2026-11-18T04:00:44Z

Content type: tutorial

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [database-consistency](<https://devfeed.tech/topics/database-consistency.md>)

Tags: [acid](<https://devfeed.tech/tags/acid.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-polaris](<https://devfeed.tech/tags/apache-polaris.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [configure](<https://devfeed.tech/tags/configure.md>), [data-engineer](<https://devfeed.tech/tags/data-engineer.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [engines](<https://devfeed.tech/tags/engines.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [getting-started](<https://devfeed.tech/tags/getting-started.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [managed](<https://devfeed.tech/tags/managed.md>), [matters](<https://devfeed.tech/tags/matters.md>), [popular](<https://devfeed.tech/tags/popular.md>), [practices](<https://devfeed.tech/tags/practices.md>), [query](<https://devfeed.tech/tags/query.md>), [service](<https://devfeed.tech/tags/service.md>)

### AI overview

Fivetran's Managed Data Lake Service uses Apache Polaris as its default Iceberg REST catalog, maintaining table metadata for Parquet data in supported cloud destinations. Query tables through the catalog so engines read committed snapshots, apply deletes, follow schema changes, and avoid hard-coded file paths; Fivetran alone can modify catalog metadata. The article gives OAuth2 setup examples for Spark, DuckDB, Snowflake, and Trino/Starburst. Requirements and capabilities differ: Spark needs Iceberg 1.9.0+ with Spark 3.4 or 3.5; DuckDB's REST support is experimental, supports SELECT and time travel, and is currently limited to S3-compatible storage; Snowflake external-table reads are generally available while writes are in preview, with configurable refresh; Trino/Starburst use REST catalog properties and vended credentials. Protect metadata files and configure refresh and snapshot retention for freshness and time-travel needs.

### Source excerpt

Why the catalog matters, how to configure it with popular query engines, and the essential best practices every data engineer should follow.

## Testing Apache Iceberg Across Dremio, Spark, Flink, and Trino

DevFeed: [Testing Apache Iceberg Across Dremio, Spark, Flink, and Trino](<https://devfeed.tech/articles/testing-apache-iceberg-across-dremio-spark-flink-and-trino-138582.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/testing-apache-iceberg-across-dremio-spark-flink-and-trino/>)

Author: Alex Merced

Published: 2026-11-17T15:23:57Z

Content type: tutorial

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [compatibility](<https://devfeed.tech/tags/compatibility.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [flink](<https://devfeed.tech/tags/flink.md>), [spark](<https://devfeed.tech/tags/spark.md>), [testing](<https://devfeed.tech/tags/testing.md>), [trino](<https://devfeed.tech/tags/trino.md>)

### AI overview

A support matrix alone cannot establish that Dremio, Spark, Flink, and Trino interoperate on a specific Iceberg setup; test the exact engine, connector, catalog, storage, and format versions together. The proposed contract checks deterministic appends and snapshot reads with row counts and checksums, then validates schema and partition evolution, deletes, retries, commit conflicts, and injected failures. Capture snapshot and manifest metadata, query results, and catalog, engine, and storage logs so mismatches can be diagnosed by logical content rather than byte-for-byte manifest equality. Separate reader and writer certification, and gate experimental v3 features independently. A passing small CI suite is only evidence for its tested path; pre-release acceptance should cover realistic data, concurrency, and failures. The excerpt gives illustrative checks and troubleshooting scenarios, but says command syntax and engine-specific behavior must be verified in the relevant documentation.

### Source excerpt

Build an Apache Iceberg compatibility contract for Dremio, Spark, Flink, and Trino covering commits, DML, evolution, deletes, time travel, and recovery. The post Testing Apache Iceberg Across Dremio, Spark, Flink, and Trino appeared first on Dremio.

## Apache Spark vs Apache Flink: which should you use?

DevFeed: [Apache Spark vs Apache Flink: which should you use?](<https://devfeed.tech/articles/apache-spark-vs-apache-flink-which-should-you-use-133225.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/spark-vs-flink>)

Author: Databricks Staff

Published: 2026-11-13T20:44:47.516244Z

Content type: comparison

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [batch](<https://devfeed.tech/tags/batch.md>), [better](<https://devfeed.tech/tags/better.md>), [choice](<https://devfeed.tech/tags/choice.md>), [data-plus-ai-foundations](<https://devfeed.tech/tags/data-plus-ai-foundations.md>), [etl](<https://devfeed.tech/tags/etl.md>), [flink](<https://devfeed.tech/tags/flink.md>), [spark](<https://devfeed.tech/tags/spark.md>), [sql](<https://devfeed.tech/tags/sql.md>), [teams](<https://devfeed.tech/tags/teams.md>), [vs](<https://devfeed.tech/tags/vs.md>)

### AI overview

Spark and Flink both support exactly-once processing, stateful operators and SQL APIs, but Spark spans batch, streaming, SQL and AI while Flink is stream-native. Spark Real-Time Mode uses a trigger on an existing Structured Streaming query to process continuously at millisecond latency, so teams can move between latency tiers without rewriting query code. The excerpt says Databricks' benchmark found Spark matched Flink on stateless work and had lower p99 latency for stateful aggregation and stream-static joins; results came from different providers and the comparison used Spark's worst run versus Flink's best, so teams should benchmark their own workloads. Flink may fit specialized cases such as very large keyed state, complex event processing, or an existing Flink platform. Real-Time Mode requires update output mode and Kafka or a row-level foreach sink; stateful operators are available on Databricks and are slated for open-source Spark 4.3, according to the excerpt.

### Source excerpt

For most teams, Apache Spark™ is the better choice. Spark runs batch ETL, SQL analytics,...

## What is Real-Time Mode in Apache Spark?

DevFeed: [What is Real-Time Mode in Apache Spark?](<https://devfeed.tech/articles/what-is-real-time-mode-in-apache-spark-136421.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/what-is-real-time-mode>)

Author: Databricks Staff

Published: 2026-11-13T07:49:12Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>)

Tags: [17](<https://devfeed.tech/tags/17.md>), [2026](<https://devfeed.tech/tags/2026.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [autoscaling](<https://devfeed.tech/tags/autoscaling.md>), [availability](<https://devfeed.tech/tags/availability.md>), [batch](<https://devfeed.tech/tags/batch.md>), [batches](<https://devfeed.tech/tags/batches.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [blocking](<https://devfeed.tech/tags/blocking.md>), [data-plus-ai-foundations](<https://devfeed.tech/tags/data-plus-ai-foundations.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [spark](<https://devfeed.tech/tags/spark.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [structured](<https://devfeed.tech/tags/structured.md>), [trigger](<https://devfeed.tech/tags/trigger.md>)

### AI overview

Apache Spark Structured Streaming Real-Time Mode processes records continuously through long-running tasks, reducing latency from seconds to milliseconds without changing the query's core code. It is suited to workloads that need sub-second responses; micro-batch or availableNow costs less when seconds or minutes suffice. On Databricks, classic compute requires Runtime 16.4 LTS or later, Photon, autoscaling and spot instances disabled, plus an enablement setting; serverless use is limited to continuous Lakeflow Spark Declarative Pipelines. Capacity must cover concurrently running tasks. Kafka sources and sinks, foreach sinks, and most operators including stateful ones are supported. Delta output requires Kafka followed by a separate micro-batch query. Monitor Real-Time Mode latency metrics rather than batch duration; the end-to-end metric excludes sink write time. Latency varies with workload and capacity. The text says Databricks has GA support, while open-source availability is rolling out; it also notes stateful support arrives in Spark 4.3.

### Source excerpt

Real-Time Mode is a trigger for Apache Spark™ Structured Streaming that processes...

## Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake

DevFeed: [Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake](<https://devfeed.tech/articles/migrating-from-apache-hudi-based-data-lake-to-apache-iceberg-based-data-lake-130716.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/migrating-from-apache-hudi-based-data-lake-to-apache-iceberg-based-data-lake/>)

Author: Nitin Kumar

Published: 2026-11-12T15:50:48Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-hudi](<https://devfeed.tech/tags/apache-hudi.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [athena](<https://devfeed.tech/tags/athena.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [based](<https://devfeed.tech/tags/based.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [files](<https://devfeed.tech/tags/files.md>), [full](<https://devfeed.tech/tags/full.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [hudi](<https://devfeed.tech/tags/hudi.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [insert](<https://devfeed.tech/tags/insert.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The excerpt presents three Hudi-to-Iceberg migration paths on AWS. In-place migration reuses Parquet files, saving rewrite time and storage but retaining the source schema, partitioning, and file layout. Iceberg's `add_files` suits a one-time conversion and requires Hudi cleanup first; for MoR, compact logs and clean old versions to avoid missing updates or duplicate rows. Apache XTable translates Hudi metadata and can sync ongoing writes, but MoR logs must be compacted before each conversion. Both approaches share data files, so avoid destructive Iceberg maintenance until cutover; `add_files` and `snapshot` also lack column statistics in Iceberg 1.9.0-1.10.0, with the issue fixed in 1.10.1. Full migration with CTAS or CREATE plus INSERT rewrites data, allowing schema and partition changes, including hidden partitioning, at higher compute, time, and storage cost. For large tables, partitioned INSERT batches support parallelism and resumability. The excerpt demonstrates AWS Glue notebooks, general S3 buckets, and Amazon S3 Tables, and recommends validating against Hudi before cutover.

### Source excerpt

Learn how to migrate transactional data lakes from Apache Hudi to Apache Iceberg on AWS. This post walks through in-place migration with the Iceberg add_files procedure and Apache XTable, plus full table migration with CTAS and INSERT, using AWS Glue notebooks and Amazon S3 Tables, then validates the results in Amazon Athena.

## How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog

DevFeed: [How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog](<https://devfeed.tech/articles/how-intuit-migrated-from-apache-hive-metastore-to-aws-glue-data-catalog-129561.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/how-intuit-migrated-from-apache-hive-metastore-to-aws-glue-data-catalog/>)

Author: Srinivasan Krishnasamy

Published: 2026-11-11T15:43:26Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [access-controls](<https://devfeed.tech/tags/access-controls.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-services](<https://devfeed.tech/tags/ai-services.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-dynamodb](<https://devfeed.tech/tags/amazon-dynamodb.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-hive](<https://devfeed.tech/tags/apache-hive.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [data-catalog](<https://devfeed.tech/tags/data-catalog.md>), [data-discovery](<https://devfeed.tech/tags/data-discovery.md>), [hive](<https://devfeed.tech/tags/hive.md>), [integration](<https://devfeed.tech/tags/integration.md>), [intuit](<https://devfeed.tech/tags/intuit.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>)

### AI overview

Intuit migrated databases, tables, and partitions from Apache Hive Metastore (HMS) to AWS Glue Data Catalog (GDC) using a phased approach: a one-time migration, ongoing synchronization in both directions, hourly parity checks, and targeted repair syncs. A DynamoDB source-of-truth flag per table prevents sync loops. HMS changes flow through Debezium CDC and Kafka, with partition events batched over 10-second windows; GDC changes use EventBridge, Lambda, Kafka, and Flink to order events before applying them to HMS. Before migrating, Intuit recommends auditing unused objects, estimating and load-testing Glue API quotas, batching writes, and using retries with backoff to manage throttling. The migration covered databases, tables, and partitions for S3-backed data; column statistics, privileges, roles, functions, and transactions were out of scope. GDC also does not support database or table renames.

### Source excerpt

How Intuit migrated its data catalog from Apache Hive Metastore to AWS Glue Data Catalog on its journey to an open, secure lakehouse. This post covers the challenges, the phased bidirectional-sync solution, and the benefits: reduced operational overhead, improved data discovery, and tighter integration with AWS analytics services.

## Cloudflare Makes Basin Serverless Analytics Generally Available

DevFeed: [Cloudflare Makes Basin Serverless Analytics Generally Available](<https://devfeed.tech/articles/cloudflare-makes-basin-serverless-analytics-generally-available-127296.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/11/cloudflare-basin-analytics/>)

Author: Renato Losio

Published: 2026-11-10T19:14:00Z

Content type: release

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [Cloudflare](<https://devfeed.tech/topics/cloudflare.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Amazon Aurora](<https://devfeed.tech/topics/amazon-aurora.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Cloudflare Workers](<https://devfeed.tech/topics/cloudflare-workers.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-ml-data-engineering](<https://devfeed.tech/tags/ai-ml-data-engineering.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [architecture-design](<https://devfeed.tech/tags/architecture-design.md>), [availability](<https://devfeed.tech/tags/availability.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloudflare](<https://devfeed.tech/tags/cloudflare.md>), [cloudflare-basin-analytics](<https://devfeed.tech/tags/cloudflare-basin-analytics.md>), [data-analytics](<https://devfeed.tech/tags/data-analytics.md>), [data-portability](<https://devfeed.tech/tags/data-portability.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [egress](<https://devfeed.tech/tags/egress.md>), [fees](<https://devfeed.tech/tags/fees.md>), [formats](<https://devfeed.tech/tags/formats.md>), [general-availability](<https://devfeed.tech/tags/general-availability.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [managed](<https://devfeed.tech/tags/managed.md>), [ml-data-engineering](<https://devfeed.tech/tags/ml-data-engineering.md>), [news](<https://devfeed.tech/tags/news.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [open-data](<https://devfeed.tech/tags/open-data.md>), [platform](<https://devfeed.tech/tags/platform.md>), [r2](<https://devfeed.tech/tags/r2.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [serverless-analytics](<https://devfeed.tech/tags/serverless-analytics.md>), [service](<https://devfeed.tech/tags/service.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

Cloudflare's Basin combines serverless data ingestion, Iceberg table management, and distributed SQL queries on R2, aiming to give smaller teams analytics without managing infrastructure. Basin Pipelines accepts events from Workers, HTTP endpoints, and Logpush, applies SQL transformations, and writes to Iceberg tables or R2; Basin Catalog manages Iceberg metadata and table maintenance; Basin SQL queries Iceberg tables. The platform supports Iceberg-compatible tools including PyIceberg, DuckDB, Snowflake, and Apache Spark, with free egress intended to enable portability across tools, regions, and clouds. Cloudflare says Basin is not a replacement for every warehouse workload; a data analyst describes it as an option for teams whose data is too large for spreadsheets but does not justify dedicated infrastructure. Since preview, Cloudflare reports higher ingestion capacity, expanded SQL capabilities, improved maintenance, Logpush integration, and Terraform support. Real-time processing, newer Iceberg specification support, adaptive maintenance, and more sovereignty and access controls are planned without committed release dates.

### Source excerpt

Cloudflare has announced the general availability of Basin, a serverless analytics platform built on Apache Iceberg and R2 Object Storage. The managed service supports open data formats and integrates with tools such as DuckDB, Snowflake, and Apache Spark, with no data egress fees. By Renato Losio

## Canonical validates Charmed Apache Spark on VMware vSphere Kubernetes

DevFeed: [Canonical validates Charmed Apache Spark on VMware vSphere Kubernetes](<https://devfeed.tech/articles/canonical-validates-charmed-apache-spark-on-vmware-vsphere-kubernetes-121216.md>)

Original publisher: [Read original article](<https://canonical.com//blog/canonical-validates-charmed-apache-spark-on-vmware-vsphere-kubernetes>)

Author: Canonical

Published: 2026-11-06T08:54:46Z

Content type: release

Language: en

Sources: [Canonical](<https://devfeed.tech/sources/canonical.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [canonical](<https://devfeed.tech/topics/canonical.md>), [VMware Cloud Foundation](<https://devfeed.tech/topics/vcf-9-1.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [automated](<https://devfeed.tech/tags/automated.md>), [automated-deployment](<https://devfeed.tech/tags/automated-deployment.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [canonical](<https://devfeed.tech/tags/canonical.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-native](<https://devfeed.tech/tags/cloud-native.md>), [data](<https://devfeed.tech/tags/data.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [data-teams](<https://devfeed.tech/tags/data-teams.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [vks](<https://devfeed.tech/tags/vks.md>), [vmware](<https://devfeed.tech/tags/vmware.md>), [vmware-cloud-foundation](<https://devfeed.tech/tags/vmware-cloud-foundation.md>), [vsphere](<https://devfeed.tech/tags/vsphere.md>)

### AI overview

Canonical says it validated Charmed Apache Spark on VMware vSphere Kubernetes Service (VKS), the Kubernetes component of VMware Cloud Foundation. The stack provides container images and CLI tooling, with Juju automating operations, to deploy, manage, and monitor Spark workloads. Its stated operational benefits include declarative cluster deployment and scaling, lifecycle management, monitoring, and security patching. Canonical says Ubuntu Pro provides continuous security updates, CVE patching, and enterprise governance under one license. The excerpt presents these capabilities as supporting cloud-native portability and efficient use of private-cloud compute; it does not provide performance measurements or describe the validation criteria.

### Source excerpt

Enterprise data teams now have further validations and support around their big data processing workloads deployed using Charmed Apache Spark on VMware vSphere Kubernetes Service (VKS), - the Kubernetes component of VMware Cloud Foundation (VCF). Today Canonical announced the company has validated Charmed Apache Spark on VKS, bringing enterprise data teams a better supported, cloud-native [...]

## Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19

DevFeed: [Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19](<https://devfeed.tech/articles/introducing-apache-spark-4-3-now-available-in-databricks-runtime-19-117923.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/introducing-apache-spark-43-now-available-databricks-runtime-19>)

Author: Jungtaek Lim; Wenchen Fan; Andreas Neumann; Serge Rielau; Hyukjin Kwon; Jerry Peng; DB Tsai; Daniel Tenedorio; Xiao Li; Reynold Xin

Published: 2026-11-05T00:00:00Z

Content type: release

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Stateful](<https://devfeed.tech/topics/stateful.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>), [apache-arrow](<https://devfeed.tech/topics/apache-arrow.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [spark](<https://devfeed.tech/tags/spark.md>), [stateful](<https://devfeed.tech/tags/stateful.md>)

### AI overview

Apache Spark 4.3 extends Structured Streaming's Real-Time Mode with stateful aggregation, deduplication and custom processing, reporting P99 end-to-end latency below 150 ms across three tested benchmarks; those measurements are not a guarantee for all workloads. It enables continuous data flow through concurrent stage scheduling and pipelined shuffles, while retaining Spark's state store and checkpointing. The release also adds Auto CDC SCD Type 2 history through Python, SQL and Spark Connect, requiring a catalog connector that supports Data Source V2 row-level operations (built-in Parquet does not). Other additions include opt-in shuffle-free execution for eligible small queries, Spark Connect session and operation visibility, JSON and VARIANT processing, and ASOF JOIN and nanosecond timestamp previews that may change. Spark 4.3 starts a quarterly feature-release cadence.

### Source excerpt

IntroductionApache Spark 4.3 brings stateful processing to Real-Time Mode, achieving...

## Scaling Model Evaluation at Pinterest with Ray (Part 4)

DevFeed: [Scaling Model Evaluation at Pinterest with Ray (Part 4)](<https://devfeed.tech/articles/scaling-model-evaluation-at-pinterest-with-ray-part-4-118874.md>)

Original publisher: [Read original article](<https://medium.com/pinterest-engineering/scaling-model-evaluation-at-pinterest-with-ray-part-4-4a96cd12cc11?source=rss----4c5a5f6279b6---4>)

Author: Pinterest Engineering

Published: 2026-11-04T16:01:04Z

Content type: article

Language: en

Sources: [Pinterest](<https://devfeed.tech/sources/pinterest-engineering-blog-medium.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Advanced AI workflows](<https://devfeed.tech/topics/advanced-ai-workflows.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>)

Tags: [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model-evaluation](<https://devfeed.tech/tags/model-evaluation.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [pinterest](<https://devfeed.tech/tags/pinterest.md>), [ray](<https://devfeed.tech/tags/ray.md>), [scaling](<https://devfeed.tech/tags/scaling.md>)

### AI overview

Pinterest describes applying Ray to model evaluation in feature importance workflows, combining Ray with Apache Spark to compute and aggregate metrics across large inference datasets. Accumulators introduced in its batch inference system support metrics such as ROC-AUC and cross entropy, calculated for each experiment and model head. The post says the platform overlaps metric computation with inference, amortizing the work and reducing added runtime cost; it reports up to a 32% reduction in end-to-end evaluation time while scaling linearly as experiments increase. The supplied text is an excerpt and does not include the implementation details behind these results.

### Source excerpt

Vishnu Arun | Tech Lead, Feature Importance Platform; Anderson Lam | Sr Software Engineer; Andrew Qiu | Machine Learning Engineer I; Helena Xu | Data Scientist I; Tianyou Wang | Data Scientist II Introduction In our multi-part blog series, we have discussed why we chose Ray as a last mile data processing framework, how we integrated Ray into our existing ML Training Infrastructure, and how we enabled efficient Batch Inference at scale. In this blog post, we will discuss how we were able to apply Ray to another critical part of our ML Infrastructure stack: Model Evaluation. In the Ray Batch Inference Blog Post, we ended on a section on accumulators (counters that would allow us to calculate metrics like ROC-AUC and cross entropy). This post is where those accumulators pay off: a model evaluation platform that combines the strengths of Ray and Apache Spark to compute metrics efficiently at scale. Model evaluation may appear to be a relatively straightforward step after inference or training. But as ML workflows scale, evaluation quickly becomes an infrastructure challenge. Systems that run many experiments in parallel, such as feature importance, must efficiently compute and aggregate metrics over large volumes of inference data, as shown in Diagram 1. Later in this post, we'll show how we amortize the metrics compute step, overlapping it with inference for minimal added runtime cost, cutting end-to-end evaluation time by up to 32% while scaling linearly as we run more experiments. Diagram 1: Depicts a feature importance system's architecture. We illustrate a multi-head model with h heads and metrics calculated per head. In this type of workflow, metrics are calculated for each experiment and model head. Note: We omit a batched view of this data found in large scale production ML systems for simplicity. Terminology Feature Importance (FI): A technique for quantifying how much each input feature contributes to a model's predictions. In this post, we will reference a pe

## How Kueue schedules Spark jobs fairly on Red Hat OpenShift AI

DevFeed: [How Kueue schedules Spark jobs fairly on Red Hat OpenShift AI](<https://devfeed.tech/articles/how-kueue-schedules-spark-jobs-fairly-on-red-hat-openshift-ai-116313.md>)

Original publisher: [Read original article](<https://developers.redhat.com/articles/2026/11/04/how-kueue-schedules-spark-jobs-fairly-on-red-hat-openshift-ai>)

Author: Vedant Deshpande

Published: 2026-11-04T13:16:25Z

Content type: article

Language: en

Sources: [Red Hat](<https://devfeed.tech/sources/red-hat.md>), [Red Hat Developer](<https://devfeed.tech/sources/red-hat-developer.md>)

Topics: [kueue](<https://devfeed.tech/topics/kueue.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>)

Tags: [admission](<https://devfeed.tech/tags/admission.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [jobs](<https://devfeed.tech/tags/jobs.md>), [kueue](<https://devfeed.tech/tags/kueue.md>), [openshift-ai](<https://devfeed.tech/tags/openshift-ai.md>), [red-hat-openshift](<https://devfeed.tech/tags/red-hat-openshift.md>), [schedules](<https://devfeed.tech/tags/schedules.md>), [spark](<https://devfeed.tech/tags/spark.md>), [spark-operator](<https://devfeed.tech/tags/spark-operator.md>)

### AI overview

Kueue controls when SparkApplications may start by suspending each submission, evaluating its declared CPU and memory against a ClusterQueue, and admitting it to the Spark Operator when quota is available. In the example, a self-managed queue reserves 3 CPUs and 4 GiB: one app with a driver and two executors fits, while a second waits without creating pods and starts after the first finishes. Setup requires the external Red Hat build of Kueue, SparkApplication framework integration and RBAC, a ResourceFlavor/ClusterQueue/LocalQueue, and a queue-name label on every app. The excerpt warns that preemption deletes driver and executor pods without checkpointing, dynamic executor allocation is unsupported, and quota reflects requested rather than actual resource use. It recommends fixed executor counts, explicit resource requests, and checking Workload conditions to distinguish a quota wait from a configuration problem.

### Source excerpt

Table of contents: How Kueue and the Spark Operator divide the work What you will build Set up the integration The post How Kueue schedules Spark jobs fairly on Red Hat OpenShift AI appeared first on Red Hat Developer.

## SpBench: An Agent-Native Framework for Cross-Engine Lakehouse Benchmarking

DevFeed: [SpBench: An Agent-Native Framework for Cross-Engine Lakehouse Benchmarking](<https://devfeed.tech/articles/spbench-an-agent-native-framework-for-cross-engine-lakehouse-benchmarking-113641.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.00449>)

Author: Lyndon Shi, Nic Crouch, Yi Hu, Zihao Ye, Choden Konigsmark, Eric Vandenberg, Dionysios Logothetis, Kuba Niemiec, Krzysztof Gadomski, Yuxiong He

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Phoronix Test Suite](<https://devfeed.tech/topics/phoronix-test-suite.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [building](<https://devfeed.tech/tags/building.md>), [cadence](<https://devfeed.tech/tags/cadence.md>), [code](<https://devfeed.tech/tags/code.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [connect](<https://devfeed.tech/tags/connect.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [custom](<https://devfeed.tech/tags/custom.md>), [engine](<https://devfeed.tech/tags/engine.md>), [framework](<https://devfeed.tech/tags/framework.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [native](<https://devfeed.tech/tags/native.md>)

### AI overview

SpBench uses an agent-native framework to make cross-engine lakehouse benchmarking repeatable and actionable. It compiles platform-agnostic benchmark definitions into comparable, platform-specific implementations, records runs as structured, queryable telemetry, and provides three composable agent skills for building benchmarks and analyzing results. In production, it benchmarks Snowpark, Snowpark Connect for Apache Spark, and other execution environments across standard suites such as TPCx-BB and TPC-DI, as well as custom workloads. The authors report that it enabled more than 400 regularly run benchmark queries, helping detect regressions whose identification directly led to production improvements.

### Source excerpt

arXiv:2611.00449v1 Announce Type: new Abstract: SpBench is an agent-native framework for cross-engine lakehouse benchmarking, in production use for benchmarking Snowpark, Snowpark Connect for Apache Spark, and other execution environments across standardized suites -- such as TPCx-BB and TPC-DI -- and custom benchmarks distilled from production workload patterns. It runs continuously across a broad portfolio of workloads, guiding the performance roadmap and detecting regressions for production-scale data engines. Benchmarking has historically been an expensive investment requiring continuous cross-functional collaboration and domain expertise, making it difficult to obtain actionable performance insights. SpBench provides (i) a typed, code-generated benchmark substrate that compiles platform-agnostic benchmark definitions into fair, comparable platform-specific implementations; (ii) a structured-telemetry contract recording every run as queryable fact and dimension data, letting LLM agents query results deterministically; and (iii) an agent-native harness of three composable skills to expedite building benchmarks and analyzing results. This agent-native architecture enabled the rapid development of more than 400 benchmark queries that run on a regular cadence to detect performance regressions; catching these has directly led to improvements in production.

## Sampling Ad Exchange Events at Scale: What We Kept, What We Dropped, and Why

DevFeed: [Sampling Ad Exchange Events at Scale: What We Kept, What We Dropped, and Why](<https://devfeed.tech/articles/sampling-ad-exchange-events-at-scale-what-we-kept-what-we-dropped-and-why-112261.md>)

Original publisher: [Read original article](<https://hackernoon.com/sampling-ad-exchange-events-at-scale-what-we-kept-what-we-dropped-and-why?source=rss>)

Author: Vikrant Bachhav

Published: 2026-11-02T06:00:04Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>)

Tags: [2019](<https://devfeed.tech/tags/2019.md>), [2022](<https://devfeed.tech/tags/2022.md>), [2023](<https://devfeed.tech/tags/2023.md>), [adtech](<https://devfeed.tech/tags/adtech.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [bi](<https://devfeed.tech/tags/bi.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [billing](<https://devfeed.tech/tags/billing.md>), [business](<https://devfeed.tech/tags/business.md>), [click](<https://devfeed.tech/tags/click.md>), [cloud-cost-optimization](<https://devfeed.tech/tags/cloud-cost-optimization.md>), [costs](<https://devfeed.tech/tags/costs.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-sampling-techniques](<https://devfeed.tech/tags/data-sampling-techniques.md>), [events](<https://devfeed.tech/tags/events.md>), [exchange](<https://devfeed.tech/tags/exchange.md>), [executor](<https://devfeed.tech/tags/executor.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [reporting](<https://devfeed.tech/tags/reporting.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [scale](<https://devfeed.tech/tags/scale.md>), [spark](<https://devfeed.tech/tags/spark.md>), [stream](<https://devfeed.tech/tags/stream.md>), [unity](<https://devfeed.tech/tags/unity.md>), [volume](<https://devfeed.tech/tags/volume.md>), [workers](<https://devfeed.tech/tags/workers.md>)

### AI overview

Unity reduced ad exchange stream volume from about 10 GB/s to 2 GB/s by sampling pre-filter events at 0.5% while retaining post-filter events at full fidelity. Sampling by auction ID kept related events joinable; counts from sampled data were scaled by the inverse sampling probability, while ratios remained comparable when both parts used the same sample. After reviewing reporting needs with report owners, the team moved reports to a separate sampled Kafka topic and tuned Spark executors. Worker count fell from 240 to 30 and CPU utilization rose 41%; DSP Report transform billing was about 41% lower for March 9-April 6, 2023 than the prior period. Two weeks of parallel comparisons preceded report cutover. Validation also uncovered longstanding join duplication that inflated BidWin and WinPrice metrics. The account notes that dimensions considered unsafe to sample were kept at full fidelity.

### Source excerpt

How we reduced Kafka stream volume by 80%, cut Spark workers from 240 to 30, and lowered reporting costs through sampling and executor tuning at Unity.

## Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks

DevFeed: [Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks](<https://devfeed.tech/articles/announcing-general-availability-of-real-time-mode-for-spark-declarative-pipelines-on-databricks-106962.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/announcing-general-availability-real-time-mode-spark-declarative-pipelines-databricks>)

Author: Jerry Peng; Abhay Bothra; Sudhanva Huruli; Navneeth Nair; Anant Pingle; Murali Talluri

Published: 2026-10-29T22:30:04Z

Content type: release

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>)

Tags: [alerting](<https://devfeed.tech/tags/alerting.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [batch](<https://devfeed.tech/tags/batch.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [data-streaming](<https://devfeed.tech/tags/data-streaming.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [declarative](<https://devfeed.tech/tags/declarative.md>), [general-availability](<https://devfeed.tech/tags/general-availability.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [operational](<https://devfeed.tech/tags/operational.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [platform](<https://devfeed.tech/tags/platform.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [spark](<https://devfeed.tech/tags/spark.md>), [time](<https://devfeed.tech/tags/time.md>), [use-cases](<https://devfeed.tech/tags/use-cases.md>)

### AI overview

Real-Time Mode (RTM) is now generally available in Databricks Spark Declarative Pipelines (SDP), enabling operational streaming pipelines with millisecond-range latency while retaining SDP's orchestration and recovery features. It streams continuously, runs query stages simultaneously, and passes records between stages as they are produced. Enable it by setting the pipeline to `continuous`, setting `spark.databricks.streaming.realTimeMode.enabled` to `true`, and defining an update flow with a `RealTime` trigger and sink target. For OLTP destinations such as Lakebase or Redis, the excerpt describes a ForEach sink with a writer implementing `open`, `process`, and `close`. Databricks reports up to 92% faster feature computation than Apache Flink on tested customer workloads, and p99 latency of 200 ms from Kafka to the online feature store for its Feature Store architecture. These are reported results, not guarantees for all workloads; the excerpt points to separate documentation for supported sources and sinks.

### Source excerpt

We introduced Real-Time Mode (RTM) last year which enabled customers to build low-latency, operational use cases...

## Apache Iceberg DML Support Matrix: MERGE, UPDATE, DELETE, and Branch Writes by Engine

DevFeed: [Apache Iceberg DML Support Matrix: MERGE, UPDATE, DELETE, and Branch Writes by Engine](<https://devfeed.tech/articles/apache-iceberg-dml-support-matrix-merge-update-delete-and-branch-writes-by-engine-102841.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/apache-iceberg-dml-support-matrix-merge-update-delete-and-branch-writes-by-engine/>)

Author: Alex Merced

Published: 2026-10-27T18:57:03Z

Content type: comparison

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [CRUD](<https://devfeed.tech/topics/crud.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [branch](<https://devfeed.tech/tags/branch.md>), [compare](<https://devfeed.tech/tags/compare.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [delete](<https://devfeed.tech/tags/delete.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [engine](<https://devfeed.tech/tags/engine.md>), [flink](<https://devfeed.tech/tags/flink.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [merge](<https://devfeed.tech/tags/merge.md>), [spark](<https://devfeed.tech/tags/spark.md>), [trino](<https://devfeed.tech/tags/trino.md>), [update](<https://devfeed.tech/tags/update.md>), [write](<https://devfeed.tech/tags/write.md>)

### AI overview

Iceberg does not define MERGE, UPDATE, or DELETE; engines and connectors provide those SQL surfaces and determine commit behavior. The text describes Spark as documenting MERGE/UPDATE/DELETE with typically copy-on-write rewrites; Trino as documenting MERGE and generally replacing affected files; Dremio as documenting MERGE/UPDATE/DELETE with source-specific branch limits; and Flink as emphasizing INSERT, overwrite, and sink-dependent upserts rather than a universal Iceberg SQL MERGE. Branch targeting and conflict handling vary by engine, connector, and version, so matching SQL does not establish equivalent semantics. The recommended validation is to test commits, concurrency, file changes, branch pointers, and reader visibility in staging, while measuring bytes written, retries, file counts, scan cost, and commit latency. Copy-on-write simplifies reads but can amplify writes; merge-on-read can reduce write IO while increasing read work and compaction needs.

### Source excerpt

Compare documented Iceberg MERGE, UPDATE, DELETE, overwrite, and branch-write support across Spark, Flink, Trino, and Dremio, with version caveats. The post Apache Iceberg DML Support Matrix: MERGE, UPDATE, DELETE, and Branch Writes by Engine appeared first on Dremio.

## Apache Iceberg Schema Evolution for Nested Data: Safe Changes and Compatibility Rules

DevFeed: [Apache Iceberg Schema Evolution for Nested Data: Safe Changes and Compatibility Rules](<https://devfeed.tech/articles/apache-iceberg-schema-evolution-for-nested-data-safe-changes-and-compatibility-rules-102842.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/apache-iceberg-schema-evolution-for-nested-data-safe-changes-and-compatibility-rules/>)

Author: Alex Merced

Published: 2026-10-27T18:56:58Z

Content type: article

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [schema design](<https://devfeed.tech/topics/schema-design.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [compatibility](<https://devfeed.tech/tags/compatibility.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [engine](<https://devfeed.tech/tags/engine.md>), [ids](<https://devfeed.tech/tags/ids.md>), [maps](<https://devfeed.tech/tags/maps.md>), [nested](<https://devfeed.tech/tags/nested.md>), [rules](<https://devfeed.tech/tags/rules.md>), [schema-evolution](<https://devfeed.tech/tags/schema-evolution.md>), [tests](<https://devfeed.tech/tags/tests.md>), [type](<https://devfeed.tech/tags/type.md>)

### AI overview

Iceberg resolves nested fields by stable field ID, which can allow metadata-only additions, drops, renames, and reordering without rewriting files when readers honor IDs. Assign new IDs to added or re-added fields; keep the ID for a rename. New fields should be optional or have defaults, and making optional fields required without defaults can break reads. Map-key type changes cannot be evolved in place and require a rewrite or a new column and migration. Type promotions vary by format version and engine. The excerpt recommends checking metadata IDs, staging changes, and testing reads across every production engine and connector version before rollout; its example reports Spark and Dremio passing while an older Trino connector returned nulls after a rename. If a reader fails compatibility checks, upgrade it, rewrite data, or migrate to a new column, then monitor errors and nulls and retain a rollback snapshot.

### Source excerpt

Evolve nested Iceberg structs, lists, and maps safely using field IDs, nullability, type-promotion rules, defaults, and cross-engine compatibility tests. The post Apache Iceberg Schema Evolution for Nested Data: Safe Changes and Compatibility Rules appeared first on Dremio.

## Apache Iceberg Clustering and Sort Order Evolution: When to Rewrite Data

DevFeed: [Apache Iceberg Clustering and Sort Order Evolution: When to Rewrite Data](<https://devfeed.tech/articles/apache-iceberg-clustering-and-sort-order-evolution-when-to-rewrite-data-102840.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/apache-iceberg-clustering-and-sort-order-evolution-when-to-rewrite-data/>)

Author: Alex Merced

Published: 2026-10-27T18:56:57Z

Content type: article

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Sorting](<https://devfeed.tech/topics/sorting.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [rewrite](<https://devfeed.tech/tags/rewrite.md>), [sort](<https://devfeed.tech/tags/sort.md>)

### AI overview

Changing an Apache Iceberg sort order updates table metadata for future writes; it does not reorder existing files or refresh their file statistics. Rewriting old files is worthwhile only when measured pruning and query savings justify the I/O, commit load, and operational risk. The text recommends identifying frequent, high-value predicates, recording baseline files read, bytes scanned, and query latency, then testing the new sort order and a bounded rewrite on representative files. Compare pruning and end-to-end performance after each batch, while monitoring commit latency, retries, storage activity, and coordinator load. If gains persist, expand in controlled waves; stop if payback is weak or operational costs exceed the measured benefit. Exact DDL and rewrite procedures depend on the Iceberg, engine, and catalog versions, so verify supported syntax and APIs in the deployed environment. The suggested batch sizes and cluster observations are examples, not universal guarantees.

### Source excerpt

Evolve Iceberg sort orders safely, measure pruning quality, choose sort or Z-order rewrites, and decide when existing data justifies a controlled rewrite. The post Apache Iceberg Clustering and Sort Order Evolution: When to Rewrite Data appeared first on Dremio.

## Apache Iceberg Write Distribution Modes: None, Hash, Range, and File Sizing

DevFeed: [Apache Iceberg Write Distribution Modes: None, Hash, Range, and File Sizing](<https://devfeed.tech/articles/apache-iceberg-write-distribution-modes-none-hash-range-and-file-sizing-102843.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/apache-iceberg-write-distribution-modes-none-hash-range-and-file-sizing/>)

Author: Alex Merced

Published: 2026-10-27T18:56:55Z

Content type: article

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [hashing](<https://devfeed.tech/topics/hashing.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [execution](<https://devfeed.tech/tags/execution.md>), [file](<https://devfeed.tech/tags/file.md>), [hash](<https://devfeed.tech/tags/hash.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [none](<https://devfeed.tech/tags/none.md>), [range](<https://devfeed.tech/tags/range.md>), [spark](<https://devfeed.tech/tags/spark.md>), [target](<https://devfeed.tech/tags/target.md>), [write](<https://devfeed.tech/tags/write.md>)

### AI overview

Iceberg write distribution controls how Spark groups rows before creating files: none adds no grouping, hash shuffles by partition columns, and range globally orders by partition and sort keys. Hash is a practical starting point for partitioned tables; none suits inputs already laid out for writing; range can improve clustering and pruning when sort keys match common queries, at the cost of extra planning, memory, and job time. The target file size is an estimate rather than a guarantee: task input, skew, and compression affect actual file sizes. The article recommends testing modes on representative data, tuning Spark shuffle partitions and adaptive execution, and checking file counts and size percentiles plus representative read performance. Verify property names and CTAS behavior for the deployed Spark and Iceberg versions.

### Source excerpt

Choose Iceberg write distribution mode none, hash, or range, then tune Spark adaptive execution and target file sizes for balanced, efficient writes. The post Apache Iceberg Write Distribution Modes: None, Hash, Range, and File Sizing appeared first on Dremio.

## Apache Polaris with Spark 4: REST Catalog, OAuth, and Credential Vending

DevFeed: [Apache Polaris with Spark 4: REST Catalog, OAuth, and Credential Vending](<https://devfeed.tech/articles/apache-polaris-with-spark-4-rest-catalog-oauth-and-credential-vending-102845.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/apache-polaris-with-spark-4-rest-catalog-oauth-and-credential-vending/>)

Author: Alex Merced

Published: 2026-10-27T18:56:45Z

Content type: tutorial

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [OAuth](<https://devfeed.tech/topics/oauth.md>), [OAuth 2.0](<https://devfeed.tech/topics/oauth2.md>), [REST API](<https://devfeed.tech/topics/rest-api.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-polaris](<https://devfeed.tech/tags/apache-polaris.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [connect](<https://devfeed.tech/tags/connect.md>), [credentials](<https://devfeed.tech/tags/credentials.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [oauth](<https://devfeed.tech/tags/oauth.md>), [rest](<https://devfeed.tech/tags/rest.md>), [scopes](<https://devfeed.tech/tags/scopes.md>), [spark](<https://devfeed.tech/tags/spark.md>), [storage](<https://devfeed.tech/tags/storage.md>), [verify](<https://devfeed.tech/tags/verify.md>)

### AI overview

The excerpt gives a versioned recipe for connecting Spark 4 to Apache Polaris 1.8 as an Iceberg REST catalog: align Polaris client JARs with the documented compatibility matrix, configure OAuth client credentials and scope, set the warehouse and vended-credential header, and map temporary credentials into the storage FileIO. A namespace and managed-table write/read smoke test checks catalog and object-store access; metadata success with S3 AccessDenied points to credentials not reaching or being applied by the FileIO. Polaris vends credentials only for physical tables managed under its storage rules, not generic Iceberg tables. The excerpt also recommends checking driver, executor, and monitoring logs for exposed OAuth secrets or temporary credentials, and validating networking and IAM permissions before a staged rollout. Exact property names and non-AWS behavior depend on the documented client and cloud configuration.

### Source excerpt

Connect Spark 4 to Apache Polaris with the Iceberg REST catalog, OAuth credentials, catalog scopes, and storage credential vending, then verify writes. The post Apache Polaris with Spark 4: REST Catalog, OAuth, and Credential Vending appeared first on Dremio.

## Built for Agents: The Case for Spark Declarative Pipelines

DevFeed: [Built for Agents: The Case for Spark Declarative Pipelines](<https://devfeed.tech/articles/built-for-agents-the-case-for-spark-declarative-pipelines-98506.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/built-agents-case-spark-declarative-pipelines>)

Author: Lisa Cao; Sudhanva Huruli

Published: 2026-10-23T19:00:00Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [Boilerplate](<https://devfeed.tech/topics/boilerplate.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [22](<https://devfeed.tech/tags/22.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-coding](<https://devfeed.tech/tags/ai-coding.md>), [ai-research](<https://devfeed.tech/tags/ai-research.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [declarative](<https://devfeed.tech/tags/declarative.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [over](<https://devfeed.tech/tags/over.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [production](<https://devfeed.tech/tags/production.md>), [sdp](<https://devfeed.tech/tags/sdp.md>), [share](<https://devfeed.tech/tags/share.md>), [spark](<https://devfeed.tech/tags/spark.md>), [write](<https://devfeed.tech/tags/write.md>)

### AI overview

In a benchmark of 22 data engineering tasks, Spark Declarative Pipelines produced nearly as many successful builds as procedural PySpark (258/264 versus 255/264), while catching structural errors in a dry run before cluster execution. Declarative builds used 17 executor-seconds versus 596 for procedural builds and had a median 64 lines of code versus 131. The approaches had similar silent logic-error rates, so tests and review remain necessary. Declarative builds used about 2.3 times as many LLM tokens--roughly $0.50 versus $0.26 per median build at Opus 4.8 list rates--but the excerpt argues this fixed upfront cost can be offset by avoiding repeated, data-dependent compute waste. Findings are from a specific benchmark, with its setup and results reportedly available in a public repository.

### Source excerpt

AI agents now write a growing share of production data pipelines (over 70% of SDP...

[Next page](<https://devfeed.tech/topics/spark.md?cursor=WyIyMDI2LTEwLTIzVDE5OjAwOjAwKzAwOjAwIiwgImZmMDA4NTA1LWIyMGEtNDVmMy1iOTViLWE3NWI3NmI0N2E5ZSJd>)