# Apache Flink

Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## A modern dilemma: Consume data from a stream or a lake

DevFeed: [A modern dilemma: Consume data from a stream or a lake](<https://devfeed.tech/articles/a-modern-dilemma-consume-data-from-a-stream-or-a-lake-141161.md>)

Original publisher: [Read original article](<https://medium.com/capital-one-tech/a-modern-dilemma-consume-data-from-a-stream-or-a-lake-296bbb354094?source=rss----3db3a67cb648---4>)

Author: Capital One Tech

Published: 2026-11-18T17:35:14Z

Content type: article

Language: en

Sources: [Capital One Tech](<https://devfeed.tech/sources/capital-one-tech.md>)

Topics: [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [etl](<https://devfeed.tech/topics/etl.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [apis](<https://devfeed.tech/tags/apis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [batch](<https://devfeed.tech/tags/batch.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [business-processes](<https://devfeed.tech/tags/business-processes.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [enrichment](<https://devfeed.tech/tags/enrichment.md>), [etl](<https://devfeed.tech/tags/etl.md>), [event](<https://devfeed.tech/tags/event.md>), [flink](<https://devfeed.tech/tags/flink.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [latency](<https://devfeed.tech/tags/latency.md>), [legacy](<https://devfeed.tech/tags/legacy.md>), [log](<https://devfeed.tech/tags/log.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [model-context-protocol](<https://devfeed.tech/tags/model-context-protocol.md>), [software-architecture](<https://devfeed.tech/tags/software-architecture.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [stream](<https://devfeed.tech/tags/stream.md>)

### AI overview

Choosing a data consumption pattern early can avoid costly architectural changes later. APIs support synchronous, record-by-record requests and paginated multirecord results, making them suitable for low-latency needs; Model Context Protocol can provide similar access for agents and is often an API wrapper. Event streams, commonly built with Kafka, Flink or Spark Streaming, suit sub-second latency and enable per-event calculations, transformations and enrichment. The excerpt also identifies Delta Change Data Feed (CDF) as a record-by-record pattern and traditional batch ETL as a multirecord pattern, but ends before explaining CDF's mechanics or comparing these approaches further.

### Source excerpt

Data is a critical component of software architecture, providing the foundation for business processes, including those powered by artificial intelligence (AI). Quality, volume, latency, consistency, ordering and duplicates are essential considerations in software design. Modern cloud architectures consume data in various ways, such as through application programming interfaces (APIs), streams and data lakes. You can consume data at the individual event or record level as it arrives or in larger batches. Legacy architectures often process data in scheduled batches, forcing surrounding architecture and business processes to depend on that approach. Getting the data consumption approach right up front is vital, because rearchitecting later is costly. General patterns to consume data There are several general data consumption patterns. These include APIs (record-by-record or multirecord results), event stream consumption (record-by-record), Delta Change Data Feed (CDF) consumption (record-by-record) or traditional batch ETL-extract, transform, load (multirecord results). APIs are ideal for low-latency, record-by-record requirements. They also support multirecord results, typically through pagination. APIs usually follow a synchronous approach, where the client makes a call and waits for a response before moving to the next step. Model Context Protocol is an industry specification for agents to consume data and can support record-by-record or batch results via pagination, similar to APIs. They are often a wrapper on top of APIs. The event stream pattern works well for sub-second latency requirements and typically runs on a tech stack like Kafka, Flink or Spark Streaming. These technologies excel at record-level calculations (like on-the-fly calculation), transformations (like column mapping) or enrichment (like adding data to an existing event). Delta Lake is often used in data lakes and serves as a transaction log to standard Apache Parquet data files (source). CDF is

## LinkedIn Cuts Daily Spark Shuffle Fetch Failures from 750 to 20 with Apache Celeborn

DevFeed: [LinkedIn Cuts Daily Spark Shuffle Fetch Failures from 750 to 20 with Apache Celeborn](<https://devfeed.tech/articles/linkedin-cuts-daily-spark-shuffle-fetch-failures-from-750-to-20-with-apache-celeborn-140877.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/11/linkedin-celeborn-remote-shuffle/>)

Author: Leela Kumili

Published: 2026-11-18T15:05:00Z

Content type: news

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [control-plane](<https://devfeed.tech/topics/control-plane.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [application](<https://devfeed.tech/tags/application.md>), [architecture-design](<https://devfeed.tech/tags/architecture-design.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [big-data-infrastructure](<https://devfeed.tech/tags/big-data-infrastructure.md>), [cloud-architecture](<https://devfeed.tech/tags/cloud-architecture.md>), [compression](<https://devfeed.tech/tags/compression.md>), [compute](<https://devfeed.tech/tags/compute.md>), [compute-costs](<https://devfeed.tech/tags/compute-costs.md>), [control-plane](<https://devfeed.tech/tags/control-plane.md>), [cost](<https://devfeed.tech/tags/cost.md>), [costs](<https://devfeed.tech/tags/costs.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [development](<https://devfeed.tech/tags/development.md>), [devops](<https://devfeed.tech/tags/devops.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [event-driven-architecture](<https://devfeed.tech/tags/event-driven-architecture.md>), [fault-tolerance](<https://devfeed.tech/tags/fault-tolerance.md>), [flink](<https://devfeed.tech/tags/flink.md>), [linkedin](<https://devfeed.tech/tags/linkedin.md>), [linkedin-celeborn-remote-shuffle](<https://devfeed.tech/tags/linkedin-celeborn-remote-shuffle.md>), [news](<https://devfeed.tech/tags/news.md>), [optimizations](<https://devfeed.tech/tags/optimizations.md>), [performance-tuning](<https://devfeed.tech/tags/performance-tuning.md>), [platform](<https://devfeed.tech/tags/platform.md>), [remote](<https://devfeed.tech/tags/remote.md>), [replication](<https://devfeed.tech/tags/replication.md>), [runtimes](<https://devfeed.tech/tags/runtimes.md>), [spark](<https://devfeed.tech/tags/spark.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

LinkedIn deployed Apache Celeborn as a shared remote shuffle service for most Spark and Flink workloads, separating shuffle storage from compute. Map tasks push intermediate data to dedicated workers, and reducers fetch it directly; this reduces contention and dependence on executor lifecycles. Compared with its previous Magnet implementation, LinkedIn reports daily shuffle fetch failures falling from about 750 to 20, compute costs down 10%, and application runtimes improved 14%. Reliability work included a five-node Raft control plane across availability zones, network-aware replication, and health-based worker exclusion; task failures during infrastructure disruptions fell 20-fold. LinkedIn also reports 40% shorter shuffle duration after moving storage to NVMe, a 35% smaller shuffle-worker fleet with Zstandard instead of LZ4, and nearly 90% lower shuffle cost per task. The deployment required substantial engineering beyond the upstream project.

### Source excerpt

LinkedIn deployed Apache Celeborn as a shared remote shuffle platform across most Spark and Flink workloads, reducing daily shuffle fetch failures from approximately 750 to 20. The deployment also lowered compute costs by 10%, improved application runtimes by 14%, and reduced shuffle costs per task by nearly 90% through dedicated storage, replication, and data-plane optimizations. By Leela Kumili

## Mitigating CVE-2026-114938 ("Java4Shell") in Apache Flink deployments

DevFeed: [Mitigating CVE-2026-114938 ("Java4Shell") in Apache Flink deployments](<https://devfeed.tech/articles/mitigating-cve-2026-114938-java4shell-in-apache-flink-deployments-141162.md>)

Original publisher: [Read original article](<https://flink.apache.org/2026/11/18/mitigating-cve-2026-114938-java4shell-in-apache-flink-deployments/>)

Published: 2026-11-18T00:00:00Z

Content type: tutorial

Language: en

Sources: [Apache Flink](<https://devfeed.tech/sources/apache-flink-stateful-computations-over-data-streams-on-apache-flink.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [cve remediation](<https://devfeed.tech/topics/cve-remediation.md>), [openjdk](<https://devfeed.tech/topics/openjdk.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [avx](<https://devfeed.tech/tags/avx.md>), [avx-512-vbmi](<https://devfeed.tech/tags/avx-512-vbmi.md>), [base64](<https://devfeed.tech/tags/base64.md>), [bug](<https://devfeed.tech/tags/bug.md>), [change](<https://devfeed.tech/tags/change.md>), [code](<https://devfeed.tech/tags/code.md>), [cve](<https://devfeed.tech/tags/cve.md>), [cve-2026-114938](<https://devfeed.tech/tags/cve-2026-114938.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [deployments](<https://devfeed.tech/tags/deployments.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [flag](<https://devfeed.tech/tags/flag.md>), [flink](<https://devfeed.tech/tags/flink.md>), [intrinsic](<https://devfeed.tech/tags/intrinsic.md>), [java4shell](<https://devfeed.tech/tags/java4shell.md>), [jdk](<https://devfeed.tech/tags/jdk.md>), [jvm](<https://devfeed.tech/tags/jvm.md>)

### AI overview

CVE-2026-114938 ("Java4Shell") is a JDK flaw in OpenJDK's AVX-512 VBMI Base64 decoding intrinsic, so mitigation requires updating the JDK used by Flink, not changing Flink code. It affects Flink processes on x86-64 CPUs with AVX-512 VBMI and affected Java versions; Java 8 and ARM are unaffected. Upgrade to the JDK vendor's fixed version or later, and move off Java releases that no longer receive updates. Until then, Java 11 and later can disable the intrinsic with `-XX:-UseBASE64Intrinsics`, configured in `env.java.opts.all`; Java 8 does not recognize the flag and will not start. Official `flink` Docker Hub images are rebuilt when maintained Temurin bases get the fix, while older tags, `apache/flink` images, and the operator image are not automatically rebuilt. For those images, use the flag or the stated image update options; operator pod flags can be set with Helm values `jvmArgs.operator` and `jvmArgs.webhook`.

### Source excerpt

CVE-2026-114938, also known as "Java4Shell", is a vulnerability in the JDK that can also affect Flink deployments. The bug is in the AVX-512 VBMI intrinsic that OpenJDK uses on x86-64 to speed up Base64 decoding, and OpenJDK has fixed it in JDK-8393212. Because the problem is in the JDK, Flink itself does not need a code change. What you need to upgrade is the JDK that Flink runs on. Until you can do that, there is a JVM flag that turns off the affected code.

## Testing Apache Iceberg Across Dremio, Spark, Flink, and Trino

DevFeed: [Testing Apache Iceberg Across Dremio, Spark, Flink, and Trino](<https://devfeed.tech/articles/testing-apache-iceberg-across-dremio-spark-flink-and-trino-138582.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/testing-apache-iceberg-across-dremio-spark-flink-and-trino/>)

Author: Alex Merced

Published: 2026-11-17T15:23:57Z

Content type: tutorial

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [compatibility](<https://devfeed.tech/tags/compatibility.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [flink](<https://devfeed.tech/tags/flink.md>), [spark](<https://devfeed.tech/tags/spark.md>), [testing](<https://devfeed.tech/tags/testing.md>), [trino](<https://devfeed.tech/tags/trino.md>)

### AI overview

A support matrix alone cannot establish that Dremio, Spark, Flink, and Trino interoperate on a specific Iceberg setup; test the exact engine, connector, catalog, storage, and format versions together. The proposed contract checks deterministic appends and snapshot reads with row counts and checksums, then validates schema and partition evolution, deletes, retries, commit conflicts, and injected failures. Capture snapshot and manifest metadata, query results, and catalog, engine, and storage logs so mismatches can be diagnosed by logical content rather than byte-for-byte manifest equality. Separate reader and writer certification, and gate experimental v3 features independently. A passing small CI suite is only evidence for its tested path; pre-release acceptance should cover realistic data, concurrency, and failures. The excerpt gives illustrative checks and troubleshooting scenarios, but says command syntax and engine-specific behavior must be verified in the relevant documentation.

### Source excerpt

Build an Apache Iceberg compatibility contract for Dremio, Spark, Flink, and Trino covering commits, DML, evolution, deletes, time travel, and recovery. The post Testing Apache Iceberg Across Dremio, Spark, Flink, and Trino appeared first on Dremio.

## Apache Iceberg Identifier Fields: Keys, CDC, Deletes, and Upserts

DevFeed: [Apache Iceberg Identifier Fields: Keys, CDC, Deletes, and Upserts](<https://devfeed.tech/articles/apache-iceberg-identifier-fields-keys-cdc-deletes-and-upserts-138578.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/apache-iceberg-identifier-fields-keys-cdc-deletes-and-upserts/>)

Author: Alex Merced

Published: 2026-11-17T15:23:36Z

Content type: article

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [identifier](<https://devfeed.tech/tags/identifier.md>), [keys](<https://devfeed.tech/tags/keys.md>)

### AI overview

Apache Iceberg identifier fields name primitive columns that identify a logical row, but they are table metadata, not an enforced primary key: uniqueness, null handling, event ordering, and deduplication remain the responsibility of writers and pipelines. Equality-delete files specify their own matching columns, which may differ from the identifier fields, so inspect them to confirm delete behavior. The excerpt describes Spark DDL for setting identifier fields and Flink upsert routing, whose exact behavior depends on connector versions. For CDC, it recommends testing duplicate and late events, checking delete files and compaction results, and preserving ordering through timestamps or sequence numbers. Before migrating to Iceberg v3 deletion vectors, verify engine and connector support and validate conversion in staging; existing files are not rewritten simply by changing identifier metadata.

### Source excerpt

Learn how Apache Iceberg identifier fields describe row identity, how engines use them for upserts and equality deletes, and what they do not guarantee. The post Apache Iceberg Identifier Fields: Keys, CDC, Deletes, and Upserts appeared first on Dremio.

## Introducing enhanced snapshots for Amazon Managed Service for Apache Flink

DevFeed: [Introducing enhanced snapshots for Amazon Managed Service for Apache Flink](<https://devfeed.tech/articles/introducing-enhanced-snapshots-for-amazon-managed-service-for-apache-flink-136180.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/introducing-enhanced-snapshots-for-amazon-managed-service-for-apache-flink/>)

Author: Francisco Morillo

Published: 2026-11-16T18:17:18Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Deployment Strategies](<https://devfeed.tech/topics/deployment-strategies.md>), [zero downtime deployment](<https://devfeed.tech/topics/zero-downtime-deployment.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-managed-service-for-apache-flink](<https://devfeed.tech/tags/amazon-managed-service-for-apache-flink.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [blue-green-deployment](<https://devfeed.tech/tags/blue-green-deployment.md>), [disaster-recovery](<https://devfeed.tech/tags/disaster-recovery.md>), [migrations](<https://devfeed.tech/tags/migrations.md>), [snapshots](<https://devfeed.tech/tags/snapshots.md>)

### AI overview

Enhanced snapshots are standalone AWS resources that let Amazon Managed Service for Apache Flink applications reuse state: start another application from a snapshot, export it to S3, or import a Flink savepoint. The text describes blue/green deployments, second-Region recovery, and migration from self-managed Flink. Blue/green cutover requires accounting for overlapping output and possible duplicates; both applications incur compute while running. Cross-Region recovery uses scheduled exports and S3 replication, trading more frequent exports for lower recovery work and higher operation, storage, and replication costs. Migration and restore require compatible Flink versions and matching operator UIDs; restores also require compatible state types and maxParallelism, and imported savepoints have validation prerequisites. Enhanced snapshots require Flink 1.20 or later, have separate IAM permissions, and persist until deleted, accruing storage charges. The text says snapshot storage is priced like application snapshots, with no added export/import charge; S3 charges still apply.

### Source excerpt

Enhanced snapshots for Amazon Managed Service for Apache Flink are now generally available. An enhanced snapshot is an AWS resource with its own ARN, decoupled from the application that created it, so you can start another application from it, export it to Amazon S3, and import savepoints for blue/green deployments, disaster recovery, and migrations.

## FlinkEval: A Synthetic Corpus for Evaluating Natural Language Interfaces to Streaming SQL

DevFeed: [FlinkEval: A Synthetic Corpus for Evaluating Natural Language Interfaces to Streaming SQL](<https://devfeed.tech/articles/flinkeval-a-synthetic-corpus-for-evaluating-natural-language-interfaces-to-streaming-sql-132705.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17093>)

Author: Alfio Massimiliano Gliozzo, Nandana Mihindukulasooriya, Gaetano Rossiello, Erick Lee, Lisa Amini

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [text2sql](<https://devfeed.tech/topics/text2sql.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [annotations](<https://devfeed.tech/tags/annotations.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [composition](<https://devfeed.tech/tags/composition.md>), [confluent](<https://devfeed.tech/tags/confluent.md>), [confluent-cloud](<https://devfeed.tech/tags/confluent-cloud.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data](<https://devfeed.tech/tags/data.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [domains](<https://devfeed.tech/tags/domains.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [execution](<https://devfeed.tech/tags/execution.md>), [interfaces](<https://devfeed.tech/tags/interfaces.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [sql](<https://devfeed.tech/tags/sql.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>)

### AI overview

FlinkEval provides 3,903 execution-grounded question-SQL pairs for evaluating natural-language interfaces to streaming SQL across 45 industry domains and 45 semantic labels. Its schemas and temporally consistent data are generated and validated iteratively; patterns from human-written pairs and vendor documentation guide quota-balanced generation, with reference queries executed and repaired on Confluent Cloud Apache Flink. The corpus covers windowing, temporal joins, pattern matching, nested types, and data manipulation, and pairs schema-explicit requests with business-language versions. A seven-level Question Specification Ladder tests reformulations and underspecified requests supplemented with oracle-derived clues. Baselines combine execution results with adjudicated LLM judgments and identify difficulties in composing patterns and specifying requests.

### Source excerpt

arXiv:2611.17093v1 Announce Type: new Abstract: Existing natural-language-to-SQL benchmarks primarily evaluate queries over bounded relational data, leaving streaming semantics underexplored. We introduce FlinkEval, an execution-grounded synthetic benchmark comprising 3,903 question--SQL pairs across 45 industry domains and 45 semantic labels. We create diverse schemas and domain-plausible, temporally consistent data through iterative generation and validation. SQL patterns collected from the human-written NL2SQL pairs and vendor documentation drive quota-balanced generation, and execution with a repair loop on Confluent Cloud Apache Flink produces reference queries covering windowing, temporal joins, pattern matching, nested types, and data manipulation. The corpus includes paired schema-explicit and business-language requests, with annotations for analysis by query pattern, complexity, and execution mode. A seven-level Question Specification Ladder examines semantics-preserving reformulation and underspecified requests augmented with oracle-derived clues. Baseline evaluations combine execution outcomes with adjudicated LLM judgments, revealing challenges in pattern composition and request specification.

## How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog

DevFeed: [How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog](<https://devfeed.tech/articles/how-intuit-migrated-from-apache-hive-metastore-to-aws-glue-data-catalog-129561.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/how-intuit-migrated-from-apache-hive-metastore-to-aws-glue-data-catalog/>)

Author: Srinivasan Krishnasamy

Published: 2026-11-11T15:43:26Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [access-controls](<https://devfeed.tech/tags/access-controls.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-services](<https://devfeed.tech/tags/ai-services.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-dynamodb](<https://devfeed.tech/tags/amazon-dynamodb.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-hive](<https://devfeed.tech/tags/apache-hive.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [data-catalog](<https://devfeed.tech/tags/data-catalog.md>), [data-discovery](<https://devfeed.tech/tags/data-discovery.md>), [hive](<https://devfeed.tech/tags/hive.md>), [integration](<https://devfeed.tech/tags/integration.md>), [intuit](<https://devfeed.tech/tags/intuit.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>)

### AI overview

Intuit migrated databases, tables, and partitions from Apache Hive Metastore (HMS) to AWS Glue Data Catalog (GDC) using a phased approach: a one-time migration, ongoing synchronization in both directions, hourly parity checks, and targeted repair syncs. A DynamoDB source-of-truth flag per table prevents sync loops. HMS changes flow through Debezium CDC and Kafka, with partition events batched over 10-second windows; GDC changes use EventBridge, Lambda, Kafka, and Flink to order events before applying them to HMS. Before migrating, Intuit recommends auditing unused objects, estimating and load-testing Glue API quotas, batching writes, and using retries with backoff to manage throttling. The migration covered databases, tables, and partitions for S3-backed data; column statistics, privileges, roles, functions, and transactions were out of scope. GDC also does not support database or table renames.

### Source excerpt

How Intuit migrated its data catalog from Apache Hive Metastore to AWS Glue Data Catalog on its journey to an open, secure lakehouse. This post covers the challenges, the phased bidirectional-sync solution, and the benefits: reduced operational overhead, improved data discovery, and tighter integration with AWS analytics services.

## Multiply pre-recorded streams with Amazon Managed Service for Apache Flink

DevFeed: [Multiply pre-recorded streams with Amazon Managed Service for Apache Flink](<https://devfeed.tech/articles/multiply-pre-recorded-streams-with-amazon-managed-service-for-apache-flink-127632.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/multiply-pre-recorded-streams-with-amazon-managed-service-for-apache-flink/>)

Author: Sivaprasad Mahamkali

Published: 2026-11-10T16:43:13Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-kinesis](<https://devfeed.tech/tags/amazon-kinesis.md>), [amazon-managed-service-for-apache-flink](<https://devfeed.tech/tags/amazon-managed-service-for-apache-flink.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [analytics-pipeline](<https://devfeed.tech/tags/analytics-pipeline.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [backpressure](<https://devfeed.tech/tags/backpressure.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [test-data](<https://devfeed.tech/tags/test-data.md>), [traffic](<https://devfeed.tech/tags/traffic.md>), [validating](<https://devfeed.tech/tags/validating.md>)

### AI overview

The stream multiplier pattern uses Amazon Managed Service for Apache Flink to amplify a pre-recorded Kinesis stream: each input yields the unchanged original plus N−1 variants. `multiple.factor` controls output volume; configured key fields stay fixed, while DataFaker generates values for selected mutable fields, preserving other sampled fields and making seeded variants repeatable. This combines observed data distributions with higher test throughput without replaying a full production-volume capture, though it does not mask sensitive fields and does not reproduce correlations among generated fields. The sample also offers a pure synthetic DataGen mode. Throughput depends on both Flink CPU and sink capacity: in the reported setup the multiplier reached about 176,000 records per second, while under-provisioned Kinesis shards throttled output. The pattern retains the sample's timing and bursts, preserves ordering only within a partition, and assumes JSON unless serializers are changed.

### Source excerpt

Validating a streaming pipeline beyond current peak takes production-scale test data. This post shows how the stream multiplier pattern on Amazon Managed Service for Apache Flink turns a small pre-recorded sample into sustained, production-scale load while preserving the statistical properties of real traffic.

## Private Continual Counting of Unbounded Streams

DevFeed: [Private Continual Counting of Unbounded Streams](<https://devfeed.tech/articles/private-continual-counting-of-unbounded-streams-126872.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2506.15018>)

Author: Ben Jacobsen, Kassem Fawaz

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [Cryptography](<https://devfeed.tech/topics/cryptography.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [absolute](<https://devfeed.tech/tags/absolute.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [alpha](<https://devfeed.tech/tags/alpha.md>), [also](<https://devfeed.tech/tags/also.md>), [art](<https://devfeed.tech/tags/art.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [based](<https://devfeed.tech/tags/based.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [but](<https://devfeed.tech/tags/but.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-ds](<https://devfeed.tech/tags/cs-ds.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [error](<https://devfeed.tech/tags/error.md>), [first](<https://devfeed.tech/tags/first.md>), [function](<https://devfeed.tech/tags/function.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [here](<https://devfeed.tech/tags/here.md>), [private](<https://devfeed.tech/tags/private.md>), [streams](<https://devfeed.tech/tags/streams.md>)

### AI overview

The paper presents a differentially private continual counting algorithm for streams whose final length is unknown. Existing matrix mechanism guarantees require parameters tuned to the input size, while the doubling trick produces suboptimal, non-smooth error. The new method uses matrix factorizations based on logarithmic perturbations of 1/√(1−z), achieving smooth error and variance O(log²(t)(log log t)^(1+2α)) for estimating the first t values, for any α > 0. It uses O(t) space and amortized O(log t) time per round. Compared with Henzinger et al.'s bounded-input method, it avoids O(n) space and O(n log n) preprocessing; optimized experiments report variance below twice theirs through t = 2³². These claims are from the supplied abstract excerpt.

### Source excerpt

arXiv:2506.15018v3 Announce Type: replace Abstract: We study the problem of differentially private continual counting in the unbounded setting where the input size $n$ is not known in advance. Current state-of-the-art algorithms based on optimal instantiations of the matrix mechanism cannot be directly applied here because their privacy guarantees only hold when key parameters are tuned to $n$. Using the common `doubling trick' avoids knowledge of $n$ but leads to suboptimal and non-smooth error. We solve this problem by introducing novel matrix factorizations based on logarithmic perturbations of the function $\frac{1}{\sqrt{1-z}}$ studied in prior works, which may be of independent interest. The resulting algorithm has smooth error, and for any $\alpha > 0$ and $t\leq n$, it is able to privately estimate the sum of the first $t$ data points with $O(\log^{2}(t)(\log\log t)^{1+2\alpha})$ variance. It requires $O(t)$ space and amortized $O(\log t)$ time per round, compared to $O(\log(n)\log(t))$ variance, $O(n)$ space and $O(n \log n)$ pre-processing time for the nearly-optimal bounded-input algorithm of Henzinger et al. Empirically, we find that our algorithm's performance is also comparable to theirs in absolute terms: with optimized parameters, our variance is less than $2\times$ theirs for $t$ as large as $2^{32}$.

## Continuous Apache Iceberg Table Maintenance with Flink: Compaction, Expiration, and Locks

DevFeed: [Continuous Apache Iceberg Table Maintenance with Flink: Compaction, Expiration, and Locks](<https://devfeed.tech/articles/continuous-apache-iceberg-table-maintenance-with-flink-compaction-expiration-and-locks-123562.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/continuous-apache-iceberg-table-maintenance-with-flink-compaction-expiration-and-locks/>)

Author: Alex Merced

Published: 2026-11-09T13:41:54Z

Content type: article

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [compaction](<https://devfeed.tech/tags/compaction.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [flink](<https://devfeed.tech/tags/flink.md>), [locks](<https://devfeed.tech/tags/locks.md>), [maintenance](<https://devfeed.tech/tags/maintenance.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>)

### AI overview

Flink TableMaintenance attaches post-commit maintenance to Iceberg tables and supports compaction, snapshot expiration, and orphan-file cleanup. Its operation depends on choosing triggers (immediate, commit-count, size, or time based), bounding each task by files, bytes, or duration, and coordinating work through a recoverable lock. The text recommends tuning thresholds against ingestion and file sizes, limiting parallelism, releasing locks between bounded tasks, and monitoring maintenance duration, lock holds, commit latency, storage requests, snapshots, and orphan files. It describes lock requirements such as persistent ownership and heartbeat state, conditional acquire and release, and safe stale-owner takeover; recovery behavior differs by implementation. Retention must exceed possible in-flight write duration, and partial work needs resume or rollback handling. Suggested limits and configuration examples are illustrative and version-sensitive: verify APIs and semantics for the target Iceberg and Flink versions, and test failover and cleanup in staging before rollout.

### Source excerpt

Run continuous Apache Iceberg maintenance with Flink using post-commit compaction, snapshot expiration, orphan cleanup, scheduling controls, and locks. The post Continuous Apache Iceberg Table Maintenance with Flink: Compaction, Expiration, and Locks appeared first on Dremio.

## Mastering stateful stream reordering at 200,000 messages per second

DevFeed: [Mastering stateful stream reordering at 200,000 messages per second](<https://devfeed.tech/articles/mastering-stateful-stream-reordering-at-200-000-messages-per-second-117179.md>)

Original publisher: [Read original article](<https://blogs.cisco.com/cisco-devnet/flink-ab-development-stateful-stream-processing>)

Author: Soumitra Kumar

Published: 2026-11-04T17:00:06Z

Content type: tutorial

Language: en

Sources: [Cisco Blogs](<https://devfeed.tech/sources/cisco-blogs.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Stateful](<https://devfeed.tech/topics/stateful.md>), [rocksdb](<https://devfeed.tech/topics/rocksdb.md>)

Tags: [amazon-eks](<https://devfeed.tech/tags/amazon-eks.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [buffer](<https://devfeed.tech/tags/buffer.md>), [checkpoint](<https://devfeed.tech/tags/checkpoint.md>), [cisco-devnet](<https://devfeed.tech/tags/cisco-devnet.md>), [devnet](<https://devfeed.tech/tags/devnet.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [exactly-once](<https://devfeed.tech/tags/exactly-once.md>), [flink](<https://devfeed.tech/tags/flink.md>), [memory](<https://devfeed.tech/tags/memory.md>), [netops](<https://devfeed.tech/tags/netops.md>), [rocksdb](<https://devfeed.tech/tags/rocksdb.md>), [stateful](<https://devfeed.tech/tags/stateful.md>), [stream](<https://devfeed.tech/tags/stream.md>)

### AI overview

At 200,000 messages per second, per-event RocksDB reads and writes plus timer registration slowed the Flink job and contributed to false positives in connection-success reporting when events arrived late or out of order. The replacement buffers each session's events as JVM heap objects in a KeyedProcessFunction, sorts by event time, and snapshots the buffer to Flink ListState at checkpoints for recovery. This removes constant state-backend access from the hot path, with the trade-off that heap use must be bounded. The team used three-minute idle eviction, timer debouncing, and compact records, then ran the new variant beside the original on production data for five months before switching default topics. The source notes that end-to-end exactly-once behavior also depends on sink handling of replayed output.

### Source excerpt

Our Flink job hit a wall at 200,000 messages per second because of RocksDB. Here is the in-memory reordering pattern that fixed it, and how A/B development proved it safe.

## Graphiant Named Confluent 2027 Data Streaming Startup of the Year for the Americas

DevFeed: [Graphiant Named Confluent 2027 Data Streaming Startup of the Year for the Americas](<https://devfeed.tech/articles/graphiant-named-confluent-2027-data-streaming-startup-of-the-year-for-the-americas-117674.md>)

Original publisher: [Read original article](<https://www.confluent.io/blog/graphiant-named-confluent-2027-data-streaming-startup-of-the-year-for-the-americas/>)

Author: Tim Graczewski

Published: 2026-11-03T21:29:59Z

Content type: news

Language: en

Sources: [Confluent: Data in motion](<https://devfeed.tech/sources/confluent-data-in-motion.md>)

Topics: [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>)

Tags: [2020](<https://devfeed.tech/tags/2020.md>), [2025](<https://devfeed.tech/tags/2025.md>), [about](<https://devfeed.tech/tags/about.md>), [adoption](<https://devfeed.tech/tags/adoption.md>), [amer](<https://devfeed.tech/tags/amer.md>), [americas](<https://devfeed.tech/tags/americas.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [confluent](<https://devfeed.tech/tags/confluent.md>), [data](<https://devfeed.tech/tags/data.md>), [data-streaming](<https://devfeed.tech/tags/data-streaming.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [network](<https://devfeed.tech/tags/network.md>), [network-observability](<https://devfeed.tech/tags/network-observability.md>), [observability](<https://devfeed.tech/tags/observability.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-time-data](<https://devfeed.tech/tags/real-time-data.md>), [real-time-data-streaming](<https://devfeed.tech/tags/real-time-data-streaming.md>), [startup](<https://devfeed.tech/tags/startup.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [uses](<https://devfeed.tech/tags/uses.md>)

### AI overview

Graphiant, named Confluent's 2027 Americas Data Streaming Startup of the Year, uses Apache Kafka on Confluent Cloud to stream telemetry and performance metrics from network edge devices. The data supports network health monitoring, traffic analysis, alerting, and customer dashboards, with analytics flowing to Apache Pinot through StarTree and to TimescaleDB. Confluent Cloud handles variable workloads as the network grows, reducing the need for Graphiant to operate Kafka infrastructure. Graphiant currently manages its own Apache Flink environment and plans to evaluate serverless Apache Flink on Confluent Cloud, as well as adopt Schema Registry to strengthen data management. The excerpt also describes a possible longer-term Streamhouse architecture and potential AI-driven network operations; these are future possibilities, not stated current capabilities.

### Source excerpt

Graphiant is Confluent's 2027 Americas Startup of the Year. Discover how it uses Apache Kafka and real-time data streaming for network observability.

## Apache Flink Agents 0.4.0 Release Announcement

DevFeed: [Apache Flink Agents 0.4.0 Release Announcement](<https://devfeed.tech/articles/apache-flink-agents-0-4-0-release-announcement-107667.md>)

Original publisher: [Read original article](<https://flink.apache.org/2026/10/30/apache-flink-agents-0.4.0-release-announcement/>)

Published: 2026-10-30T00:00:00Z

Content type: release

Language: en

Sources: [Apache Flink](<https://devfeed.tech/sources/apache-flink-stateful-computations-over-data-streams-on-apache-flink.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Release notes](<https://devfeed.tech/topics/release-notes.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [supervisor agent pattern](<https://devfeed.tech/topics/supervisor-agent-pattern.md>), [business logic](<https://devfeed.tech/topics/business-logic.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [announce](<https://devfeed.tech/tags/announce.md>), [announcement](<https://devfeed.tech/tags/announcement.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [community](<https://devfeed.tech/tags/community.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [events](<https://devfeed.tech/tags/events.md>), [execution](<https://devfeed.tech/tags/execution.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [flink](<https://devfeed.tech/tags/flink.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [processing](<https://devfeed.tech/tags/processing.md>), [production](<https://devfeed.tech/tags/production.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [release](<https://devfeed.tech/tags/release.md>)

### AI overview

Apache Flink Agents 0.4.0 adds sub-agent delegation, a unified Decision Model API for TypeSafe, OpenAI, and Alibaba Cloud Model Studio, model routing through the Java API, multimodal messages, and broader structured output support. New `ctx.chat` actions combine model calls with business logic; conditional event triggers, tool parameter injection, and Java asynchronous execution on JDK 11 and 17 extend application control. Trace Log records execution steps and call lineage, while new metrics cover processing, queues, latency, and component calls. This remains a preview, and APIs and configuration may change. Upgrading from 0.3 requires broad migration work: Action State Store data is incompatible; retries now default to zero; message, event, tool, and logging formats have changed; and stricter validation applies to settings and inputs. Pending Action snapshot restoration may fail, so validate representative state before upgrading.

### Source excerpt

The Apache Flink community is excited to announce the release of Apache Flink Agents 0.4.0. Apache Flink Agents brings AI agents into Flink streaming pipelines. Applications can continuously respond to real-time events, combine AI processing with business logic, and use Flink for distributed execution and fault recovery. Shaped by feedback from early production adopters and advances in the agent ecosystem, Flink Agents 0.4 marks the final planned step toward a production-ready 1.

## Apache Iceberg DML Support Matrix: MERGE, UPDATE, DELETE, and Branch Writes by Engine

DevFeed: [Apache Iceberg DML Support Matrix: MERGE, UPDATE, DELETE, and Branch Writes by Engine](<https://devfeed.tech/articles/apache-iceberg-dml-support-matrix-merge-update-delete-and-branch-writes-by-engine-102841.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/apache-iceberg-dml-support-matrix-merge-update-delete-and-branch-writes-by-engine/>)

Author: Alex Merced

Published: 2026-10-27T18:57:03Z

Content type: comparison

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [CRUD](<https://devfeed.tech/topics/crud.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [branch](<https://devfeed.tech/tags/branch.md>), [compare](<https://devfeed.tech/tags/compare.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [delete](<https://devfeed.tech/tags/delete.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [engine](<https://devfeed.tech/tags/engine.md>), [flink](<https://devfeed.tech/tags/flink.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [merge](<https://devfeed.tech/tags/merge.md>), [spark](<https://devfeed.tech/tags/spark.md>), [trino](<https://devfeed.tech/tags/trino.md>), [update](<https://devfeed.tech/tags/update.md>), [write](<https://devfeed.tech/tags/write.md>)

### AI overview

Iceberg does not define MERGE, UPDATE, or DELETE; engines and connectors provide those SQL surfaces and determine commit behavior. The text describes Spark as documenting MERGE/UPDATE/DELETE with typically copy-on-write rewrites; Trino as documenting MERGE and generally replacing affected files; Dremio as documenting MERGE/UPDATE/DELETE with source-specific branch limits; and Flink as emphasizing INSERT, overwrite, and sink-dependent upserts rather than a universal Iceberg SQL MERGE. Branch targeting and conflict handling vary by engine, connector, and version, so matching SQL does not establish equivalent semantics. The recommended validation is to test commits, concurrency, file changes, branch pointers, and reader visibility in staging, while measuring bytes written, retries, file counts, scan cost, and commit latency. Copy-on-write simplifies reads but can amplify writes; merge-on-read can reduce write IO while increasing read work and compaction needs.

### Source excerpt

Compare documented Iceberg MERGE, UPDATE, DELETE, overwrite, and branch-write support across Spark, Flink, Trino, and Dremio, with version caveats. The post Apache Iceberg DML Support Matrix: MERGE, UPDATE, DELETE, and Branch Writes by Engine appeared first on Dremio.

## Live Adaptation of Dataflow Stream Processing for Hybrid Edge-Cloud Analytics

DevFeed: [Live Adaptation of Dataflow Stream Processing for Hybrid Edge-Cloud Analytics](<https://devfeed.tech/articles/live-adaptation-of-dataflow-stream-processing-for-hybrid-edge-cloud-analytics-101622.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.34658>)

Author: Yuanli Wang, Prateek Jain, Lei Huang, Le Xu, Vasiliki Kalavri

Published: 2026-10-27T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [applications](<https://devfeed.tech/tags/applications.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [business-analytics](<https://devfeed.tech/tags/business-analytics.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [data](<https://devfeed.tech/tags/data.md>), [datacenter](<https://devfeed.tech/tags/datacenter.md>), [dataflow](<https://devfeed.tech/tags/dataflow.md>), [devices](<https://devfeed.tech/tags/devices.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [edge](<https://devfeed.tech/tags/edge.md>), [execution](<https://devfeed.tech/tags/execution.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [live](<https://devfeed.tech/tags/live.md>), [processing](<https://devfeed.tech/tags/processing.md>), [stream](<https://devfeed.tech/tags/stream.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>)

### AI overview

Sluice adapts streaming computations between edge devices and cloud resources as network and edge conditions change, without pausing execution. Its automatic query rewriter divides applications into independent fault-tolerant segments; live routing shifts their computations while preserving event-time semantics. The system is designed to work with existing dataflow engines and is implemented on Apache Flink. Evaluation across five real-world applications used Raspberry Pi devices and a simulated 100-VM testbed; the abstract reports higher throughput than baseline systems, but gives no numerical results or specific baseline details.

### Source excerpt

arXiv:2610.34658v1 Announce Type: new Abstract: Dataflow streaming systems, like Apache Flink and Spark Streaming, are integral to modern business analytics, providing real-time insights across industries. However, for emerging hybrid edge-cloud analytics pipelines, neither processing everything in the cloud nor static edge offloading can reliably sustain the target rate under dynamic WAN conditions and limited edge resources. In this paper, we explore how these datacenter-centric platforms can be extended to support such hybrid analytics pipelines. Our goal is to maintain full transparency for user applications, while retaining built-in fault tolerance, out-of-order processing, and existing ecosystem integrations. Achieving this is challenging because existing dataflow engines assume homogeneous resources, rely on static task placement, and do not support adaptation under dynamic network and edge conditions. We introduce Sluice, an adaptive data stream processing system for hybrid clouds, compatible with dataflow streaming engines. Sluice's key contributions are: (i) an automatic query rewriter that decomposes applications into independent, fault-tolerant segments, and (ii) a live routing mechanism that shifts computations between edge and cloud without pausing execution while preserving event-time semantics. We implement Sluice on Apache Flink and evaluate it on a physical testbed with Raspberry Pi devices, as well as on a large simulated testbed of 100 VMs. Across five real-world applications, our results show that Sluice seamlessly shifts computations between edge and cloud, while consistently achieving higher throughput than baseline systems.

## Pinterest Uses Event Time Watermarks to Finalize Petabyte Scale Data Partitions

DevFeed: [Pinterest Uses Event Time Watermarks to Finalize Petabyte Scale Data Partitions](<https://devfeed.tech/articles/pinterest-uses-event-time-watermarks-to-finalize-petabyte-scale-data-partitions-89988.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/10/pinterest-event-time-partition/>)

Author: Leela Kumili

Published: 2026-10-19T14:37:00Z

Content type: news

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Temporal data](<https://devfeed.tech/topics/temporal-data.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>)

Tags: [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [architecture-design](<https://devfeed.tech/tags/architecture-design.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-partitioning](<https://devfeed.tech/tags/data-partitioning.md>), [development](<https://devfeed.tech/tags/development.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [event-driven-architecture](<https://devfeed.tech/tags/event-driven-architecture.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [news](<https://devfeed.tech/tags/news.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>), [pinterest-event-time-partition](<https://devfeed.tech/tags/pinterest-event-time-partition.md>), [published](<https://devfeed.tech/tags/published.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

Pinterest's ingestion framework combines Apache Flink event-time progress with Apache Iceberg metadata to publish a monotonic partition-finalization watermark for continuously ingested change data. The signal helps downstream jobs determine when data is sufficiently complete, while percentile-based policies can trade completeness for freshness.

### Source excerpt

Pinterest has introduced event time-based partition finalization for its database ingestion framework. The mechanism uses Apache Flink to track event time and Apache Iceberg metadata to publish a non-regressing watermark, helping downstream jobs determine when continuously ingested CDC data is sufficiently complete for processing. By Leela Kumili

## Data Engineering Weekly #292

DevFeed: [Data Engineering Weekly #292](<https://devfeed.tech/articles/data-engineering-weekly-292-89969.md>)

Original publisher: [Read original article](<https://www.dataengineeringweekly.com/p/data-engineering-weekly-292>)

Author: Ananth Packkildurai

Published: 2026-10-19T03:58:54Z

Content type: article

Language: en

Sources: [Data Engineering Weekly](<https://devfeed.tech/sources/data-engineering-weekly.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [data observability](<https://devfeed.tech/topics/data-observability.md>), [RDF (Resource Description Framework)](<https://devfeed.tech/topics/rdf.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [Amazon Bedrock Knowledge Bases](<https://devfeed.tech/topics/amazon-bedrock-knowledge-bases.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [airbnb](<https://devfeed.tech/tags/airbnb.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-arrow](<https://devfeed.tech/tags/apache-arrow.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-bedrock](<https://devfeed.tech/tags/aws-bedrock.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>)

### AI overview

This issue of Data Engineering Weekly collects reports and resources on data infrastructure. Topics include Zephon's deterministic online execution for AI training data, Apache Arrow's columnar computing ecosystem, observability ontology at Netflix, Spark and Flink shuffle scaling at LinkedIn, and data ingestion, search, and surveillance systems at other companies.

### Source excerpt

The Weekly Data Engineering Newsletter

## TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads

DevFeed: [TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads](<https://devfeed.tech/articles/timeseries-tiered-storage-journey-kafka-flink-streams-to-native-cassandra-cold-reads-87090.md>)

Original publisher: [Read original article](<https://netflixtechblog.com/timeseries-tiered-storage-journey-kafka-flink-streams-to-native-cassandra-cold-reads-e59d597c9d60?source=rss----2615bd06b42e---4>)

Author: Netflix Technology Blog

Published: 2026-10-14T19:56:08Z

Content type: article

Language: en

Sources: [Netflix](<https://devfeed.tech/sources/netflix-techblog-medium.md>)

Topics: [Apache Cassandra](<https://devfeed.tech/topics/cassandra.md>), [cost-optimization](<https://devfeed.tech/topics/cost-optimization.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Data integrity in software engineering](<https://devfeed.tech/topics/data-integrity-in-software-engineering.md>), [Amazon Elastic Block Store (Amazon EBS)](<https://devfeed.tech/topics/amazon-elastic-block-store-amazon-ebs.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [abstraction](<https://devfeed.tech/tags/abstraction.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [api](<https://devfeed.tech/tags/api.md>), [audits](<https://devfeed.tech/tags/audits.md>), [backups](<https://devfeed.tech/tags/backups.md>), [based](<https://devfeed.tech/tags/based.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [cases](<https://devfeed.tech/tags/cases.md>), [cassandra](<https://devfeed.tech/tags/cassandra.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [flink](<https://devfeed.tech/tags/flink.md>), [journey](<https://devfeed.tech/tags/journey.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [native](<https://devfeed.tech/tags/native.md>), [storage](<https://devfeed.tech/tags/storage.md>), [streams](<https://devfeed.tech/tags/streams.md>)

### AI overview

Netflix describes two ways to move older, immutable TimeSeries data from Cassandra hot storage to S3 while retaining the same query interface. The first streams data through Kafka and Flink, then compacts it into sorted Parquet files with small row groups and cached Bloom filters; it reports roughly 150 ms p50 and 500 ms p99 reads, but adds streaming infrastructure and operational cost. The later design reads directly from existing Cassandra backups. A ReadOnlyCompactionStrategy reduces SSTable overlap before slices become immutable; Cassandra's pluggable Memtables then use backup metadata and locally cached index components to plan parallel, cached S3 chunk reads. Retries, integrity checks, initialization isolation, and cluster-wide checks for completed compaction and uploaded backups protect the transition. Because S3 reads use more CPU, they can run on a separate, smaller Cassandra cluster. The excerpt reports over 15 PB stored, thousands of cold reads per second per region, about 30% better p90 latency than the earlier cold tier, and p99 below 500 ms.

### Source excerpt

By Oleksii Tkachuk, Rajiv Shringi, Chris Lohfink, Abe Ratnofsky, Kartik Sathyanarayanan Introduction In an earlier post, we introduced the TimeSeries Abstraction, our high-throughput, low-latency platform designed to store and query temporal event data at scale. It has since become the operational backbone for Netflix's largest datasets, which now span multiple petabytes. However, managing such large datasets in hot storage, tuned for low-millisecond latency access, is not cheap. At Netflix, cost-efficiency is a continuous discipline rather than a temporary initiative. This post shares two solutions that significantly reduce storage costs by continuously moving older, less frequently accessed data into cheaper object storage: Solution 1: Uses Apache Kafka and Flink to stream data directly to S3 Solution 2: Leverages Cassandra backups in S3 and pluggable Memtables to read data directly from S3 We'll start with the Kafka/Flink-based design, then describe how we evolved to a native Cassandra-based cold tier. Challenges As Netflix's TimeSeries datasets have expanded, we've encountered several compounding challenges: Growing retention demands: Users often push for increasing data retention to serve a broader range of use cases, which in turn increases the storage costs. Escalating costs: Maintaining massive, dense clusters for infrequently accessed historical data is prohibitively expensive. Sparse historical access: Most queries target recent events, but older data must remain readily available for occasional audits and investigations. Operational strain: High storage density on hot nodes complicates scaling, deployment, and cluster stability. Latency requirements: Even historical data sometimes requires sub-second query performance for internal tooling and analysis. This creates a clear opportunity: keep the most recent data on fast storage and migrate older, rarely read data to cheaper storage while preserving a consistent API. Done well, this can meet user and operat

## Behind the Scenes: Evolving Netflix's Ads Event Pipeline for Live -- Part II

DevFeed: [Behind the Scenes: Evolving Netflix's Ads Event Pipeline for Live -- Part II](<https://devfeed.tech/articles/behind-the-scenes-evolving-netflix-s-ads-event-pipeline-for-live-part-ii-87081.md>)

Original publisher: [Read original article](<https://netflixtechblog.com/behind-the-scenes-evolving-netflixs-ads-event-pipeline-for-live-part-ii-826ebf9ad9fb?source=rss----2615bd06b42e---4>)

Author: Netflix Technology Blog

Published: 2026-10-14T19:51:35Z

Content type: article

Language: en

Sources: [Netflix](<https://devfeed.tech/sources/netflix-techblog-medium.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [join](<https://devfeed.tech/tags/join.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [netflix](<https://devfeed.tech/tags/netflix.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [processing](<https://devfeed.tech/tags/processing.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [security](<https://devfeed.tech/tags/security.md>), [security-practices](<https://devfeed.tech/tags/security-practices.md>)

### AI overview

Netflix rebuilt its Ads Event Publisher around Apache Flink stateful stream processing, replacing a database lookup with a real-time join of ad-serving responses and client events. The change removes a registry write from the ad-serving critical path, supports live-stream traffic, and adds in-stream deduplication and recovery for late events. The article also describes cross-region routing and a gradual rollout that compared the new and legacy pipelines.

### Source excerpt

By Yogesh Nagarur Introduction In Part 1 of this series, we shared how Netflix built the Ads Event pipeline and approached it as a system design problem: replacing fragmented pipelines with a centralized collection, enrichment, and a standard data contract. The result was Ads Event Publisher, the system that collects ad telemetry from client devices, enriches it with the context of the ad that was actually served, and publishes a single, unified stream of ad events to every downstream consumer: frequency capping, metrics, billing, measurement partners, and reporting. Any member viewing or account details that move through this pipeline end-to-end are encrypted and handled in accordance with Netflix's strict privacy and security practices to safeguard member PII. If ad serving is the "brain" and ad events are the "heartbeats" that feed it, then enrichment is what gives each heartbeat its meaning, turning a bare "[Somebody] watched an ad to completion" signal into "this campaign, this creative, at this price, for this break." Part 1 described that enrichment step as a single box in the architecture. This post is about what was inside that box, and why we rebuilt it. Part 1 ended with a look at what was on the horizon: ads on live streams, de-duplication, and richer data signals. A single project delivered all three by rebuilding the Ads Event Publisher around stateful stream processing. We replaced a database lookup with a real-time streaming join in Apache Flink, which let the system scale for Live, took a dependency off the critical path, and replaced the engine of a high-traffic pipeline without downtime. Let's dive in. A quick recap: enrichment and the Metadata Registry When an ad is served, the ad server knows everything about it: the campaign, the creative, the price of the impression, the targeting context, and the tracking URLs that measurement partners expect us to fire. The client device, by contrast, knows almost nothing: just an opaque token and when to se

## Filter, reshape, and redact logs before export with Transformers

DevFeed: [Filter, reshape, and redact logs before export with Transformers](<https://devfeed.tech/articles/filter-reshape-and-redact-logs-before-export-with-transformers-85609.md>)

Original publisher: [Read original article](<https://blog.cloudflare.com/transformers-on-logpush/>)

Author: Sahidya Devadoss

Published: 2026-10-12T14:44:59Z

Content type: article

Language: en

Sources: [Cloudflare Blog](<https://devfeed.tech/sources/cloudflare-blog.md>)

Topics: [telemetry](<https://devfeed.tech/topics/telemetry.md>), [apache-arrow](<https://devfeed.tech/topics/apache-arrow.md>), [IO](<https://devfeed.tech/topics/io.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>)

Tags: [batching](<https://devfeed.tech/tags/batching.md>), [cdn](<https://devfeed.tech/tags/cdn.md>), [cgo](<https://devfeed.tech/tags/cgo.md>), [data](<https://devfeed.tech/tags/data.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [data-volume](<https://devfeed.tech/tags/data-volume.md>), [developers](<https://devfeed.tech/tags/developers.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

Cloudflare Logpush Transformers let teams filter, redact, and reshape log records with SQL before export. Cloudflare embeds Apache DataFusion in its delivery service and uses Apache Arrow to move batches between Go and Rust. Transformations run in-process, and failures stop untransformed data from being delivered.

### Source excerpt

Logpush now supports Transformers, allowing teams to filter, redact, and reshape log streams using SQL before export. Powered by Apache DataFusion and Apache Arrow, Transformers apply those changes directly to log data before it reaches the destination.

## Apache Flink CDC 3.7.0 Release Announcement

DevFeed: [Apache Flink CDC 3.7.0 Release Announcement](<https://devfeed.tech/articles/apache-flink-cdc-3-7-0-release-announcement-84444.md>)

Original publisher: [Read original article](<https://flink.apache.org/2026/10/10/apache-flink-cdc-3.7.0-release-announcement/>)

Published: 2026-10-10T08:00:00Z

Content type: release

Language: en

Sources: [Apache Flink](<https://devfeed.tech/sources/apache-flink-stateful-computations-over-data-streams-on-apache-flink.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [announcement](<https://devfeed.tech/tags/announcement.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [flink-cdc](<https://devfeed.tech/tags/flink-cdc.md>), [python](<https://devfeed.tech/tags/python.md>), [release](<https://devfeed.tech/tags/release.md>), [sql](<https://devfeed.tech/tags/sql.md>), [sql-server](<https://devfeed.tech/tags/sql-server.md>)

### AI overview

Apache Flink CDC 3.7.0 adds an OpenAI-compatible model client and built-in AI functions for YAML transforms, alongside inline Python UDFs, new SQL Server and Fluss source connectors, expanded schema handling, and connector reliability fixes. The release also adds ordered asynchronous transform execution to improve throughput for I/O-bound operations while preserving data and schema event ordering.

### Source excerpt

The Apache Flink Community is excited to announce the release of Flink CDC 3.7.0! This release brings AI-powered data processing to Flink CDC pipelines with a pluggable model API, an OpenAI-compatible model client, built-in text and multimodal functions, and ordered asynchronous transform execution. It also introduces inline Python UDFs, new SQL Server and Fluss Pipeline source connectors, broader YAML Transform functions, and safer schema handling for existing sink tables. In addition, this release improves pipeline configuration and routing, and delivers reliability fixes for various connectors.

## Apache Iceberg 1.12.0: What's New, Breaking Changes, and Upgrade Guide

DevFeed: [Apache Iceberg 1.12.0: What's New, Breaking Changes, and Upgrade Guide](<https://devfeed.tech/articles/apache-iceberg-1-12-0-what-s-new-breaking-changes-and-upgrade-guide-62554.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/apache-iceberg-1-12-0-whats-new-breaking-changes-and-upgrade-guide/>)

Author: Alex Merced

Published: 2026-09-30T15:27:16Z

Content type: release

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [engine](<https://devfeed.tech/tags/engine.md>), [flink](<https://devfeed.tech/tags/flink.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [practical](<https://devfeed.tech/tags/practical.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [rest](<https://devfeed.tech/tags/rest.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [upgrade](<https://devfeed.tech/tags/upgrade.md>)

### AI overview

Apache Iceberg 1.12.0 broadens support for version 3 features across engines, tightens REST catalog requirements, improves streaming reliability, and adds groundwork for format version 4. The release also removes support for Spark 3.4 and Flink 2.0, deletes deprecated APIs, and changes position delete file writing. The article outlines production implications and an upgrade validation plan across multiple engines and shared tables.

### Source excerpt

Apache Iceberg 1.12.0 expands v3 engine support, REST catalog operations, streaming reliability, and v4 groundwork. Learn the breaking changes and use a practical upgrade plan. The post Apache Iceberg 1.12.0: What's New, Breaking Changes, and Upgrade Guide appeared first on Dremio.

## From 40 seconds to under 10: rebuilding incident detection on OpenTelemetry, Apache Kafka, and Apache Flink on Kubernetes

DevFeed: [From 40 seconds to under 10: rebuilding incident detection on OpenTelemetry, Apache Kafka, and Apache Flink on Kubernetes](<https://devfeed.tech/articles/from-40-seconds-to-under-10-rebuilding-incident-detection-on-opentelemetry-apache-kafka-and-apache-flink-on-kubernetes-62396.md>)

Original publisher: [Read original article](<https://www.cncf.io/blog/2026/09/30/from-40-seconds-to-under-10-rebuilding-incident-detection-on-opentelemetry-apache-kafka-and-apache-flink-on-kubernetes/>)

Author: Deepak Biswas, Senior Engineering Manager, Central Monitoring and Disaster Recovery, Atlassian

Published: 2026-09-30T11:00:00Z

Content type: article

Language: en

Sources: [Cloud Native Computing Foundation](<https://devfeed.tech/sources/cloud-native-computing-foundation.md>)

Topics: [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [CoreDNS Kubernetes](<https://devfeed.tech/topics/coredns-kubernetes.md>)

Tags: [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [blog](<https://devfeed.tech/tags/blog.md>), [incident](<https://devfeed.tech/tags/incident.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>)

### AI overview

The article describes rebuilding a SaaS incident detection platform with Apache Kafka, Apache Flink on Kubernetes, and OpenTelemetry. It reports that event-to-metric latency in the prior system exceeded 40 seconds, while recall improved from about 60% to a peak of 86% before falling to 64%; precision remains below the team's goal.

### Source excerpt

Every SaaS company has the same uncomfortable question after a major incident: who noticed first, the monitoring or the customers? For a long time our honest answer was "it depends". This post describes how a small...

[Next page](<https://devfeed.tech/topics/apache-flink.md?cursor=WyIyMDI2LTA5LTMwVDExOjAwOjAwKzAwOjAwIiwgIjJjNWM5OTE0LThlZTYtNDZmMy1hYWRjLTQwNDMxMzk3NWFlMyJd>)