# Apache Iceberg Open table format for large analytic datasets that enables multiple compute engines to safely use the same tables. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Run DuckDB analytics on your Amazon DynamoDB data with zero-ETL DevFeed: [Run DuckDB analytics on your Amazon DynamoDB data with zero-ETL]() Original publisher: [Read original article]() Author: Lee Hannigan Published: 2026-09-11T14:53:57Z Content type: tutorial Language: en Sources: [AWS Database Blog]() Topics: [Databases](), [Amazon S3](), [Apache Iceberg](), [AWS Lambda](), [AWS IAM](), [Latency]() Tags: [advanced-300](), [amazon-dynamodb](), [amazon-s3-tables](), [analytics](), [apache-iceberg](), [aws-lambda](), [duckdb](), [how-to](), [iam](), [integration](), [sql](), [technical-how-to]() ### AI overview This tutorial explains how to run ad hoc SQL analytics on Amazon DynamoDB data with DuckDB through a zero-ETL replication flow. ### Source excerpt Run ad hoc SQL analytics on your Amazon DynamoDB data with DuckDB. A zero-ETL integration replicates your table into Apache Iceberg tables on Amazon S3 Tables, and an AWS Lambda function running DuckDB serves SQL queries through an IAM-authorized function URL. ## Unifying governance across engines and catalogs in the Open Lakehouse DevFeed: [Unifying governance across engines and catalogs in the Open Lakehouse]() Original publisher: [Read original article]() Author: Daniel Weeks; Ryan Blue; Andrei Tserakhau Published: 2026-09-10T15:05:10Z Content type: article Language: en Sources: [Databricks]() Topics: [Apache Iceberg](), [data](), [databricks](), [Apache Spark](), [DuckDB]() Tags: [apache](), [apis](), [catalog](), [data](), [databricks](), [duckdb](), [governance](), [open](), [product](), [spark]() ### AI overview The article examines two additions to the Apache Iceberg REST Catalog specification: read restrictions and catalog labels. It explains how they support governance across catalogs and engines, including centralized and delegated enforcement, and discusses trust requirements for engines such as Spark, DuckDB, and Trino. ### Source excerpt In our previous posts, we showed how open table formats, open APIs and unified governance... ## Rerouting the Stream: How Lyft Moved to the Apache Flink Operator DevFeed: [Rerouting the Stream: How Lyft Moved to the Apache Flink Operator]() Original publisher: [Read original article]() Author: Maheep Myneni Published: 2026-08-31T19:08:16Z Content type: article Language: en Sources: [Lyft Engineering - Medium]() Topics: [Apache Iceberg](), [Streaming](), [Kubernetes](), [Open Source](), [autoscaling](), [Tooling](), [Architecture & Design]() Tags: [apache](), [apache-flink](), [autoscaling](), [data-platforms](), [infrastructure](), [kubernetes](), [lyft](), [open-source](), [production](), [streaming](), [streaming-data-processing](), [tooling]() ### AI overview Lyft describes its migration from an internally developed Flink Kubernetes operator to the open-source Apache Flink Kubernetes Operator. The move addressed maintenance burden, technical debt, feature gaps, outdated dependencies, and growing real-time streaming demands, while providing capabilities such as autoscaling, memory tuning, safer upgrades, and automatic rollbacks. ### Source excerpt Written by Maheep Myneni, Arda Kuyumcu, and Prem Santosh Udaya Shankar at Lyft. Why We Migrated: Technical Debt Meets Modern Streaming Demands Over the past several quarters, Lyft's Streaming Compute team retired our internally developed Flink Kubernetes operator and moved our entire streaming fleet onto the open-source Apache Flink Kubernetes operator. This post is about why we made the switch, how we pulled it off incrementally without disrupting users, and the follow-on work it took to actually get the benefits we were after. Back in 2020, when we first architected the Lyft Flink Kubernetes Operator, it was exactly what we needed. At that time, the open-source community hadn't yet built a dedicated control plane, so we built our own to manage all streaming applications on Kubernetes. It worked well for our initial workloads, but as our real-time data needs increased and our engineers' scope of ownership grew in both breadth and complexity, the cracks started to show. First came the maintenance burden. Our operator had become a relic of Lyft's early Kubernetes days, kept alive by a growing pile of custom code. Every Flink version upgrade meant carefully picking through layers of accumulated technical debt and hoping nothing broke on the way through. Second came the feature gap. Streaming tooling kept moving, and our engineers kept asking for capabilities that had become table stakes elsewhere, such as autoscaling to right-size jobs, automatic rollbacks on failed deploys, and an end to hand-tuning CPU and memory. Each request left us with two options, neither of which was ideal. We could explain why we couldn't support it yet, or spend weeks rebuilding something the open-source community had already shipped. Third was the dependency problem. We were pinned to outdated libraries. That doesn't break anything today, but it almost always creates new issues down the line. Security patches lagged, modern Kubernetes features stayed out of reach, and every quarter we waite ## AWS Glue 6.0 now available with 30% lower price and full Apache Iceberg v3 support DevFeed: [AWS Glue 6.0 now available with 30% lower price and full Apache Iceberg v3 support]() Original publisher: [Read original article]() Author: Channy Yun (윤석찬) Published: 2026-08-21T18:53:26Z Content type: release Language: en Sources: [AWS News Blog]() Topics: [Apache Iceberg](), [Amazon Web Services](), [Apache Spark](), [Python](), [Scala](), [Geographic Information System](), [Internet of things]() Tags: [analytics](), [apache](), [apache-iceberg](), [aws](), [aws-glue](), [launch](), [news](), [performance](), [pricing](), [python-3-13](), [real-time](), [release](), [scala](), [spark](), [streaming]() ### AI overview AWS Glue 6.0 is generally available with 30% lower pricing, a modernized Spark 4.1 runtime, Python 3.13 and Scala 2.13 support, and full Apache Iceberg v3 support. The release adds improved handling of semi-structured data, declarative ETL pipelines, faster PySpark execution, and real-time streaming with single-digit millisecond latency. ### Source excerpt AWS Glue 6.0 is built on a fully modernized runtime, Apache Spark 4.1, Python 3.13, and Scala 2.13, delivering 30% lower pricing than previous AWS Glue versions. ## Data Engineering Weekly #282 DevFeed: [Data Engineering Weekly #282]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-10T01:21:26Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [data-platforms](), [data observability](), [Graphs](), [semantic-layer](), [Large Language Model](), [Apache Iceberg](), [gRPC](), [Concurrency](), [Caching](), [Optimization](), [Streaming]() Tags: [caching](), [chaos](), [concurrency](), [data](), [data-engineering](), [data-platforms](), [data-quality](), [knowledge-graphs](), [llms](), [newsletter](), [observability](), [pipelines](), [semantic-layer](), [streaming]() ### AI overview Data Engineering Weekly #282 is a newsletter covering data platform fundamentals, semantic layers, ontology-backed knowledge graphs, converged databases, AI modernization, and Netflix's real-time distributed graph query architecture. It highlights composable architectures, data quality and observability, evolving schemas supported by LLM-assisted extraction, Iceberg full-text search, and optimization techniques including concurrency control, streaming filters, and caching. ### Source excerpt The Weekly Data Engineering Newsletter ## How Jump Trading uses ClickHouse with Iceberg for analytics DevFeed: [How Jump Trading uses ClickHouse with Iceberg for analytics]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-07-30T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Apache Iceberg](), [log management](), [data](), [Latency](), [Open Source](), [real-time](), [Database]() Tags: [analytics](), [apache](), [apache-iceberg](), [business](), [clickhouse](), [data](), [database](), [industry](), [latency](), [logging](), [logs](), [low-latency](), [observability](), [open](), [open-source](), [performance](), [real-time](), [scale](), [technology](), [trading]() ### AI overview Jump Trading uses a self-managed ClickHouse platform to capture and analyze petabyte-scale financial trading logs. The platform ingests hundreds of terabytes daily with a sub-20-second p99 and supports real-time analytics across hundreds of billions of events. To support large-scale batch reporting and research without affecting the real-time cluster, Jump added a parallel Apache Iceberg pipeline. ### Source excerpt Jump Trading captures petabyte-scale financial trading logs on a self-managed ClickHouse platform, where zero data loss and low latency are critical requirements. ## 10X more data, same 4 seconds: single-query scaling in Redpanda SQL on 1TB DevFeed: [10X more data, same 4 seconds: single-query scaling in Redpanda SQL on 1TB]() Original publisher: [Read original article]() Author: Marcin Grzebieluch Published: 2026-07-29T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [SQL](), [Benchmark](), [dataset](), [Apache Iceberg](), [Streaming]() Tags: [apache](), [benchmark](), [dataset](), [kafka](), [sql](), [streaming]() ### AI overview This article benchmarks how a single analytical query in Redpanda SQL scales as the dataset grows from 100 GB to 1 TB and as cluster resources increase. Redpanda SQL combines live-streaming topics with historical Apache Iceberg tables through bridge queries, and the benchmark uses skewed NYC Taxi trip data to evaluate strong scaling and query feasibility. ### Source excerpt A benchmark on how a single analytical query behaves in Redpanda SQL as the dataset and the cluster grow. ## How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes DevFeed: [How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes]() Original publisher: [Read original article]() Author: Amit Prabhu Published: 2026-07-14T14:06:16Z Content type: article Language: en Sources: [Razorpay Engineering - Medium]() Topics: [data](), [airflow](), [Apache Spark](), [Amazon S3](), [Apache Iceberg](), [microservices architecture](), [parquet](), [Architecture & Design]() Tags: [airflow](), [analytics](), [apache-iceberg](), [architecture](), [batch](), [data](), [graphs](), [infrastructure](), [parquet](), [pipelines](), [razorpay](), [spark](), [trino](), [warehouse]() ### AI overview Razorpay describes its data warehouse refresh pipeline, which builds wide denormalized Facts by joining data from multiple microservices. The article covers the original Airflow- and Spark-based full-refresh process, the underlying lake formats and query layer, and the scaling challenges that led the team to reconsider refresh strategy, data layout, and high-cardinality dimensions. ### Source excerpt Contributors: Utkarsh Koppikar Rohan Background Razorpay provides the payment infrastructure for millions of merchants globally. Behind every payment, settlement, and refund is a microservices architecture where each service owns its own database. While this keeps services independent and scalable, it creates a challenge for stakeholders who need to see across those boundaries. The Data Platform team manages the infrastructure that bridges this gap. Transactional data flows into the lake via CDC pipelines, ingested onto S3 in Delta Lake, Apache Iceberg, or plain Parquet formats. On top of the lake, we build domain-specific warehouse tables -- wide, pre-joined tables that co-locate all the data a consumer needs, queryable via Trino. These power two use cases: Analytics (internal dashboards on Tableau and Superset) and Reporting (merchants and regulated entities who download structured data exports; Razorpay generates nearly a million such reports per month). The warehouse tables that power both use cases are called Facts. A Fact is a flat denormalised table on S3, produced by joining 10 to 30 microservice tables and materialising the result once. A settlement Fact, for example, merges payments, refunds, adjustments, and card details into a single wide row so that a dashboard or report reads from a single table instead of joining across services in real time. It is closer to a domain-specific materialised view than a classical data warehouse fact table. We maintain over 50 such Facts, and approximately 40% of all merchant reports are served directly from them. As data volumes and the number of entities per fact grew, the batch generation pipeline began to show its limits, prompting us to rethink the refresh strategy, the data layout, and how to handle high-cardinality dimensions. The rest of this post covers that journey. The Full Refresh Pipeline: Our Baseline and the Pain The original full-refresh pipeline was straightforward. Schedule: Airflow schedules Spark jobs o ## Full CDC semantics land in the Iceberg output for Redpanda Connect DevFeed: [Full CDC semantics land in the Iceberg output for Redpanda Connect]() Original publisher: [Read original article]() Author: Prakhar Garg Published: 2026-07-14T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Apache Iceberg](), [clickhouse](), [Database](), [Kafka]() Tags: [analytics](), [clickhouse](), [data](), [database](), [engineering](), [product](), [real-time](), [redpanda-connect](), [streams](), [sync]() ### AI overview This hands-on article explains how Redpanda Connect v4.99.0 applies full change data capture semantics to Apache Iceberg tables. Its output supports row-level inserts, upserts, and deletes, allowing a lakehouse table to stay synchronized with a source database in real time through a single pipeline definition. ### Source excerpt Full CDC semantics in Iceberg mean the lakehouse reflects what the source database looks like right now, not what it looked like during last night's batch window. Get to know the latest addition to Redpanda Connect. ## Evaluating Cloudflare as a Data Platform DevFeed: [Evaluating Cloudflare as a Data Platform]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-13T12:39:09Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [Cloudflare](), [data](), [Apache Iceberg]() Tags: [apache-iceberg](), [cloudflare](), [data](), [data-platform]() ### AI overview The article examines Cloudflare's data products and whether they can support a complete data stack, including how to build with them and what they cost. The supplied excerpt establishes the article's scope but does not provide its final findings. ### Source excerpt new kid on the block ## Tableflow: Turn Kafka Topics into Iceberg Tables DevFeed: [Tableflow: Turn Kafka Topics into Iceberg Tables]() Original publisher: [Read original article]() Author: Mohtasham Sayeed Mohiuddin Published: 2026-07-10T15:36:14Z Content type: tutorial Language: en Sources: [Confluent: Data in motion]() Topics: [Apache Iceberg](), [Kafka](), [schema-evolution](), [Confluent Cloud](), [parquet](), [Amazon S3](), [Streaming](), [AWS Glue](), [Amazon Redshift](), [BigQuery]() Tags: [amazon-redshift](), [amazon-s3](), [apache-iceberg](), [aws-glue](), [bigquery](), [confluent-cloud](), [kafka](), [parquet](), [schema-evolution](), [streaming](), [technologies](), [tutorial]() ### AI overview This tutorial explains how Confluent Cloud Tableflow continuously materializes Apache Kafka topics as Apache Iceberg or Delta Lake tables. It covers automatic schema handling, type conversion, schema evolution, Parquet conversion, catalog publishing, and table maintenance for querying streaming data with analytics engines and warehouses. ### Source excerpt Learn how Confluent Tableflow turns Kafka topics into Iceberg tables for zero-ETL analytics with automatic schema evolution and open catalog access. ## Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption DevFeed: [Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption]() Original publisher: [Read original article]() Author: Rahul Penti Published: 2026-07-10T00:00:00Z Content type: article Language: en Sources: [Grab Tech]() Topics: [Apache Iceberg](), [data](), [Amazon S3](), [parquet](), [Apache Spark](), [Latency](), [Concurrency]() Tags: [apache-iceberg](), [compute](), [concurrency](), [data](), [database](), [engineering](), [latency](), [parquet](), [performance](), [s3](), [scale](), [spark]() ### AI overview Grab describes its transition from Hive Parquet and a directory-based Hive Metastore architecture toward Apache Iceberg and a table-centric lakehouse. The article outlines scalability problems involving metadata latency, small files, S3 request overhead, and operational management, and mentions a unified Spark catalog built to hide table-format differences. ### Source excerpt Introduction: The evolution of Grab's Data Lake At Grab's scale, managing petabytes of data across billions of S3 objects demands more than a storage layer. It demands a robust architectural primitive that supports the high-concurrency needs of a modern "Lakehouse." Our goal is full storage-compute separation, leveraging S3 as an elastic foundation for both near-real-time metrics and large-scale batch transformations. For years, the vast majority of our tables were Hive Parquet, managed through the Hive Metastore with a directory-based layout. This model served us well, but as data volume grew, the directory-and-metastore approach became the limiting factor. We are now transitioning to a table-centric architecture built on modern table formats, treating data as a first-class primitive to ensure consistency and performance across our internal data transformation platforms: Slide, which powers batch transformations, and Hugo, which handles online-to-data-lake ingestion. Along the way, we also built the UnifiedSparkCatalog, a unified Spark catalog that hides table-format differences from users entirely, which we are open-sourcing alongside this post. The catalyst for change: Challenges with Hive Parquet For years, Hive Parquet was the backbone of our Data Lake, representing the vast majority of our tables. However, as data volume scaled, the architectural limitations of directory-based storage became apparent. We identified four primary bottlenecks: Catalog latency: The Hive Metastore (HMS) became a centralized failure point. High concurrency during metadata access led to O(n) listing overhead, where query planning time scaled linearly with partition count, crippling throughput. The small file problem: The directory layout left us with severe file fragmentation. Certain Machine Learning (ML) datasets had an average file size under 1 MB, with thousands of files in each partition. At this scale, the overhead of S3 object listing and metadata request latency drove up Appl ## Automated Schema Evolution in Pinterest's Next-Generation DB Ingestion Framework DevFeed: [Automated Schema Evolution in Pinterest's Next-Generation DB Ingestion Framework]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-06-24T18:01:01Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [schema-evolution](), [Apache Iceberg](), [Automation](), [Databases](), [Apache Spark](), [Kafka](), [Pull Request](), [Code generation](), [datasets](), [monitor](), [Provisioning]() Tags: [automation](), [code-generation](), [datasets](), [engineering](), [icebergs](), [kafka](), [monitor](), [pinterest](), [process](), [schema-evolution](), [software](), [spark](), [storage]() ### AI overview Pinterest describes an automated schema-evolution framework for its CDC-based database ingestion platform, which uses Kafka, Flink, Spark, and Iceberg. The framework propagates supported schema changes across the stack, uses pull-request-based versioning and auditing, targets SLA-based consistency between online and offline schemas, and provides recovery paths for unsupported or ambiguous changes. ### Source excerpt Yisheng Zhou | Software Engineer II Liang Mou | Sr Staff Software Engineer Gabriel Raphael Garcia Montoya | Staff Software Engineer Istvan Podor | Staff Software Engineer Introduction In the first post of this series, we introduced Pinterest's next-generation CDC-based ingestion platform built on Kafka, Flink, Spark, and Iceberg. In production, upstream schemas are constantly evolving, and in a distributed CDC pipeline, schema is not just metadata -- it is a cross-system contract spanning ingestion, transformation, storage, and historical backfill. A schema change that is not handled carefully can break Flink jobs, block Spark upserts, or create inconsistencies between online and offline representations. This post walks through how we make schema evolution safe in practice: the onboarding model it builds on, the changes we support and the tradeoffs we accept, how updates propagate across the stack, and how we monitor and recover when things go wrong. We end with where we're heading next -- zero-gap schema evolution. Background & Motivation Schema evolution was particularly challenging in our environment because our ingestion pipeline is composed of multiple tightly related stages, each of which depends on schema in a different way. A single table onboarding flow spans CDC source configuration, Kafka provisioning, Flink and Spark code generation, Iceberg CDC and base table creation, and historical bootstrap -- all driven by the same schema. Once a source schema changes, that change must be reflected across code, metadata, and storage in a controlled and auditable way. Without a unified automation framework, manual updates across these layers increase the risk of drift, failed deployments, and inconsistent datasets. We therefore needed a way to make schema evolution safe, repeatable, and scalable. Our Solution: SLA-Based Automated Schema Evolution To address this problem, we built an automated schema evolution framework on top of our CDC ingestion platform, providing: au ## Bridge Queries in Redpanda SQL DevFeed: [Bridge Queries in Redpanda SQL]() Original publisher: [Read original article]() Author: Paul Wilkinson Published: 2026-06-23T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Apache Iceberg](), [SQL](), [Streaming](), [parquet](), [Compression](), [real-time](), [data](), [dashboards]() Tags: [analytics](), [compression](), [data](), [engineering](), [parquet](), [real-time](), [sql](), [storage](), [streaming]() ### AI overview This article introduces bridge queries in Redpanda SQL, which combine historical data from Iceberg tables with recent messages from Redpanda topics through a single virtual SQL table. By reading the freshness gap directly from the topic, teams can flush data to Iceberg less frequently, producing better-sized Parquet files while retaining near-real-time query results and reducing compaction overhead. ### Source excerpt Stop choosing between fresh data and robust Parquet files. Redpanda SQL bridge queries let you query live streaming topics and historical Iceberg tables together, without the compaction overhead. ## June 2026 newsletter DevFeed: [June 2026 newsletter]() Original publisher: [Read original article]() Author: Mark Needham Published: 2026-06-18T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [data](), [Open Source](), [real-time](), [wasm](), [WebAssembly](), [Apache Iceberg](), [Parser]() Tags: [analytics](), [clickhouse](), [data](), [database](), [engineering](), [open-source](), [performance](), [real-time](), [release](), [systems](), [wasm](), [webassembly]() ### AI overview The June 2026 ClickHouse newsletter marks the project's tenth anniversary, highlights its open-source community, and covers contributions to WebAssembly user-defined functions, geospatial and Iceberg functionality, and performance improvements in ClickHouse 26.5. ### Source excerpt Welcome to the June 2026 ClickHouse newsletter, which will round up what's happened in real-time data warehouses over the last month. ## The open ecosystem around ClickHouse DevFeed: [The open ecosystem around ClickHouse]() Original publisher: [Read original article]() Author: Al Brown Published: 2026-06-17T15:41:30Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Model Context Protocol](), [Open Source](), [OpenTelemetry](), [Apache Iceberg](), [Database](), [MCP Server](), [observability](), [Prometheus](), [schema-evolution](), [Artificial Intelligence]() Tags: [ai](), [clickhouse](), [database](), [mcp-server](), [model-context-protocol](), [observability](), [open](), [open-source](), [opentelemetry](), [prometheus](), [schema-evolution]() ### AI overview ClickHouse's open ecosystem connects databases with observability, AI tooling, and lakehouse technologies through open standards. The article highlights OpenTelemetry and Prometheus support, an open-source MCP server compatible with multiple agent frameworks, and support for Iceberg and Delta Lake formats. ### Source excerpt Around any good database sits the clients you import, the dashboards your team shares, the pipelines that feed it, the projects that build on top of it, and now the agents that query it. ## Announcing Apache Iceberg 1.11.0 DevFeed: [Announcing Apache Iceberg 1.11.0]() Original publisher: [Read original article]() Author: Google Open Source (noreply@blogger.com) Published: 2026-05-27T18:30:00Z Content type: release Language: en Sources: [Google Open Source Blog]() Topics: [Apache Iceberg](), [Apache Spark](), [apache-flink](), [schema-evolution](), [Streaming](), [Encryption](), [Cryptography]() Tags: [apache-flink](), [apache-iceberg](), [apache-spark](), [encryption](), [kms](), [lakehouse](), [releases](), [schema-evolution](), [streaming]() ### AI overview Apache Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, makes both default build targets, and introduces changes including server-side REST catalog scan planning, partition statistics APIs, built-in envelope encryption with Google KMS support, and Google Storage Analytics integration. ### Source excerpt by Alex Stephen & Talat Uyarer, Lakehouse Apache Iceberg project has just launched version 1.11.0! A lot has happened since the last version. Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, the latest releases of the two engines and makes both the default build targets The rest are more structural. The REST catalog learns to plan scans server-side, shifting metadata work off the query engine. A new partition statistics scan API gives optimizers a clean, supported way to read a table's shape. Built-in table encryption arrives with envelope encryption and Google KMS support. And Google Storage Analytics library integration makes your Iceberg workloads faster than before. Let's take a look at some of the biggest changes. Spark & Flink Updates As Spark and Flink are moving forward, the 1.11.0 release is pushing forward for new version support in both. Spark 4.1 & DSv2 Migration: Spark 4.1 unlocks is MERGE INTO with automatic schema evolution: Spark's newer MERGE syntax accepts a WITH SCHEMA EVOLUTION clause, so a MERGE whose source carries columns the target table lacks can add those columns to the table within the same statement, with no separate ALTER TABLE round trip. Beyond the version bump, the 1.11 Spark connector also modernizes against Spark's newer DataSource V2 APIs and adds an asynchronous micro-batch planner that speeds up Structured Streaming. Flink Ecosystem Updates: Initial work for Flink 2.1 support has landed in the core repository, continuing Iceberg's promise of providing first-class, low-latency streaming sink capabilities. The centerpiece of the Flink work is the DynamicIcebergSink, an experimental sink that breaks the old one-sink-per-table model: a single sink routes each record to a table chosen at runtime, creating tables on demand and evolving their schemas and partition specs on the fly as the input changes including dropping columns once you opt in with dropUnusedColumns. In addition to DynamicIcebergSInk work Flink sta ## Redpanda SQL is GA: the query engine that skips the pipeline DevFeed: [Redpanda SQL is GA: the query engine that skips the pipeline]() Original publisher: [Read original article]() Author: Matt Schumpert Published: 2026-05-27T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Query (disambiguation)](), [Apache Iceberg](), [data-engineering](), [Streaming](), [Kafka]() Tags: [data-engineering](), [migration](), [redpanda-byoc](), [sql](), [streaming]() ### AI overview Redpanda SQL is a Postgres-based query engine available inside Redpanda BYOC clusters. It lets teams query live-streaming topics and historical Apache Iceberg tables in one SQL statement without an ETL pipeline or connector fleet, reducing ingestion delays and visibility gaps. ### Source excerpt Your warehouse can't query data that hasn't been ingested yet. Redpanda SQL can. Ad hoc SQL against live topics and Iceberg history, no ETL pipeline required. ## How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33% DevFeed: [How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%]() Original publisher: [Read original article]() Author: Nick Del Nano, Data Streaming Published: 2026-05-21T00:00:00Z Content type: article Language: en Sources: [Yelp]() Topics: [data](), [data lake](), [Apache Iceberg](), [Amazon S3](), [AWS IAM]() Tags: [analytics](), [apache-iceberg](), [aws](), [data](), [data-lake](), [iam](), [partition](), [s3]() ### AI overview Yelp describes visualizations that map partition keys against access-event timestamps to reveal daily batch jobs, backfills, and ad hoc queries. The resulting usage attribution supported Apache Iceberg migration and storage-efficiency work that reduced the cost of its petabyte-scale data lake by 33%. ### Source excerpt Introduction In large analytics environments, data teams often struggle to answer deceptively simple questions, like who their stakeholders are and how their data is being used. At Yelp, we address this by visualizing access patterns, plotting time-based partition key values against access event timestamps. These visualizations reveal distinct usage signatures - ad hoc queries, daily batch jobs, and periodic backfills - allowing data owners to understand their stakeholders and use cases. This deeper insight into data usage has enabled high-impact platform initiatives including migrating thousands of tables to Apache Iceberg format and identifying storage efficiencies which reduced the cost of... ## ClickHouse at Google Cloud Next '26 DevFeed: [ClickHouse at Google Cloud Next '26]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-04-27T11:34:34Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Google](), [Cloud](), [cpu](), [Benchmark](), [Apache Iceberg](), [data](), [real-time](), [cloud-infrastructure]() Tags: [announcement](), [arm](), [benchmark](), [clickhouse](), [cloud](), [data](), [google](), [google-cloud](), [integrations](), [migration](), [open](), [partnership](), [performance](), [platform](), [product](), [real-time](), [speed](), [storage](), [systems]() ### AI overview ClickHouse at Google Cloud Next '26 recaps ClickHouse Cloud running on Google Axion ARM-based processors and a native integration with Google's Lakehouse Runtime Catalog for querying Iceberg tables in Google Cloud Storage. The article reports top ClickBench results, faster query performance, reduced data-load times, and direct querying without data movement or ETL maintenance. ### Source excerpt Google Cloud Next is one of the biggest moments in the cloud calendar, and this year ClickHouse made a splash. Not just with a booth, house party and some swag, but a wave of product launches, integrations, and a deepening partnership with Google Cloud. ## ClickHouse integrates with Lakehouse Runtime Catalog DevFeed: [ClickHouse integrates with Lakehouse Runtime Catalog]() Original publisher: [Read original article]() Author: Melvyn Peignon Published: 2026-04-22T13:27:18Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Apache Iceberg](), [Google](), [BigQuery](), [Apache Spark](), [Cloud](), [API](), [data](), [Authentication]() Tags: [analytics](), [apache-iceberg](), [api](), [authentication](), [bigquery](), [clickhouse](), [cloud](), [data](), [google](), [google-cloud](), [integration](), [spark]() ### AI overview ClickHouse announces a beta integration with Google's Lakehouse Runtime Catalog, enabling direct querying of Google Cloud Lakehouse Apache Iceberg tables through the Iceberg REST Catalog. The integration supports ClickHouse 26.2 and later, ClickHouse Cloud, Google Application Default Credentials, and querying or loading data without data movement or metadata synchronization. ### Source excerpt We're excited to announce a new integration between ClickHouse and Google's Lakehouse Runtime CatalogCatalog ## ClickHouse is data lake ready DevFeed: [ClickHouse is data lake ready]() Original publisher: [Read original article]() Author: Karolina Ruiz Rogelj; Melvyn Peignon Published: 2026-03-25T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Apache Iceberg](), [parquet](), [Data Space](), [real-time](), [Cloud](), [Amazon S3](), [Azure]() Tags: [analytics](), [apache-iceberg](), [azure](), [caching](), [clickhouse](), [cloud](), [concurrency](), [data](), [interoperability](), [open-source](), [parquet](), [performance](), [real-time](), [s3](), [scale](), [storage]() ### AI overview ClickHouse is presented as data lake ready, with direct querying of Iceberg and Delta Lake data, support for cloud catalogs and Parquet optimizations, and the option to load data into native storage for faster high-concurrency analytics while writing results back to open formats. ### Source excerpt ClickHouse now supports direct querying of Iceberg and Delta Lake formats across major cloud catalogs without requiring data migration. ## Introducing Iceberg output for Redpanda Connect DevFeed: [Introducing Iceberg output for Redpanda Connect]() Original publisher: [Read original article]() Author: Prakhar Garg Published: 2026-03-05T00:00:00Z Content type: release Language: en Sources: [Redpanda]() Topics: [Apache Iceberg](), [Redpanda-Connect](), [Streaming](), [Routing (disambiguation)](), [schema-evolution](), [YAML](), [data]() Tags: [apache-iceberg](), [connectors](), [integration](), [pii](), [redpanda-connect](), [routing](), [schema-evolution](), [streaming](), [yaml]() ### AI overview Redpanda introduces an Iceberg output for Redpanda Connect that writes streaming data directly to Apache Iceberg tables through declarative YAML pipelines. The component supports transformations, enrichment, PII removal, and content-based routing across Redpanda Connect inputs and processors, enabling data from sources such as HTTP webhooks, Postgres CDC streams, and GCP Pub/Sub to reach lakehouse tables with less infrastructure and configuration overhead. ### Source excerpt Land any data into Apache Iceberg with Redpanda Connect. Maximize infra density, automate schema evolution, and scale 1-to-many routing without the toil. ## Databricks is abstracting away physical data engineering controls DevFeed: [Databricks is abstracting away physical data engineering controls]() Original publisher: [Read original article]() Author: Zach Wilson Published: 2026-02-24T01:03:11Z Content type: opinion Language: en Sources: [DataExpert.io Newsletter]() Topics: [databricks](), [data-engineering](), [Apache Iceberg](), [Open Source]() Tags: [architecture](), [data-engineering](), [databricks](), [open-source](), [partition](), [partitioning](), [sorting](), [spark]() ### AI overview This opinion article argues that Databricks is shifting away from hands-on data engineering by abstracting physical data modeling through features such as liquid clustering and predictive optimization. It also criticizes Databricks' support for managed Apache Iceberg tables after acquiring Tabular. ### Source excerpt Databricks abstracts away almost all of the data engineering skills. Liquid clustering is the first place where things will get messy! [Next page]()