# stream-processing

Stream processing is a computing paradigm and method for continuously ingesting, processing, and analyzing data as it arrives rather than in batches.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Continuous Prompts: LLM-Augmented Pipeline Processing over Unstructured Streams

DevFeed: [Continuous Prompts: LLM-Augmented Pipeline Processing over Unstructured Streams](<https://devfeed.tech/articles/continuous-prompts-llm-augmented-pipeline-processing-over-unstructured-streams-145730.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2512.03389>)

Author: Shu Chen, Deepti Raghavan, U\\u{g}ur \\c{C}etintemel

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [batching](<https://devfeed.tech/tags/batching.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-reasoning](<https://devfeed.tech/tags/llm-reasoning.md>), [long-running](<https://devfeed.tech/tags/long-running.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [operator](<https://devfeed.tech/tags/operator.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [probing](<https://devfeed.tech/tags/probing.md>), [queries](<https://devfeed.tech/tags/queries.md>), [rag](<https://devfeed.tech/tags/rag.md>), [stream](<https://devfeed.tech/tags/stream.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>), [streams](<https://devfeed.tech/tags/streams.md>)

### AI overview

Continuous Prompts (CPs) brings persistent LLM reasoning to stream processing by extending retrieval-augmented generation to streaming data and defining continuous semantic operators. Implemented in VectraFlow, it offers multiple implementations, mainly LLM-based plus an embedding-based variant. Tuple batching and operator fusion improve efficiency at the cost of accuracy. To manage that trade-off, CPs uses lightweight shadow executions and cost-aware multi-objective Bayesian optimization to learn throughput-accuracy frontiers and adapt plans within probing budgets. The abstract reports evaluations with operator microbenchmarks and real-dataset pipelines, finding that VectraFlow adapts to workload changes and sustains semantic queries over evolving unstructured streams.

### Source excerpt

arXiv:2512.03389v2 Announce Type: replace Abstract: Monitoring unstructured streams increasingly requires persistent, semantics-aware computation, yet today's LLM frameworks remain stateless and one-shot, limiting their usefulness for long-running analytics. We introduce Continuous Prompts (CPs), the first framework that brings LLM reasoning into continuous stream processing. CPs extend RAG to streaming settings, define continuous semantic operators, and provide multiple implementations, primarily focusing on LLM-based approaches but also reporting one embedding-based variants. Furthermore, we study two LLM-centric optimizations, tuple batching and operator fusion, to significantly improve efficiency while managing accuracy loss. Because these optimizations inherently trade accuracy for speed, we present a dynamic optimization framework that uses lightweight shadow executions and cost-aware multi-objective Bayesian optimization (MOBO) to learn throughput-accuracy frontiers and adapt plans under probing budgets. We implement CPs in the VectraFlow stream processing system. Using operator-level microbenchmarks and streaming pipelines on real datasets, we show that VectraFlow can adapt to workload dynamics, navigate accuracy-efficiency trade-offs, and sustain persistent semantic queries over evolving unstructured streams.

## Locality-Aware Scheduling for Stream Processing Applications in Serverless Environments

DevFeed: [Locality-Aware Scheduling for Stream Processing Applications in Serverless Environments](<https://devfeed.tech/articles/locality-aware-scheduling-for-stream-processing-applications-in-serverless-environments-142973.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23879>)

Author: Tianyu Qi, Maria A. Rodriguez, Rajkumar Buyya

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [scheduling](<https://devfeed.tech/topics/scheduling.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>)

Tags: [balance](<https://devfeed.tech/tags/balance.md>), [chaining](<https://devfeed.tech/tags/chaining.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [function](<https://devfeed.tech/tags/function.md>), [function-as-a-service](<https://devfeed.tech/tags/function-as-a-service.md>), [i-o](<https://devfeed.tech/tags/i-o.md>), [latency](<https://devfeed.tech/tags/latency.md>), [local](<https://devfeed.tech/tags/local.md>), [locality](<https://devfeed.tech/tags/locality.md>), [monitors](<https://devfeed.tech/tags/monitors.md>), [network](<https://devfeed.tech/tags/network.md>), [operator](<https://devfeed.tech/tags/operator.md>), [operators](<https://devfeed.tech/tags/operators.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [routing](<https://devfeed.tech/tags/routing.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>)

### AI overview

LcSched targets a locality tradeoff in stateful serverless stream pipelines: keeping each operator's state with its execution can force chained invocations to cross workers, raising latency and I/O overhead. The distributed strategy aims to run chained invocations locally while ensuring they access state on the same worker, and monitors runtime workload to balance workers. The abstract reports up to 3.6x higher throughput and up to 79% lower end-to-end latency than the state-of-the-art distributed scheduling strategy for stateful serverless environments.

### Source excerpt

arXiv:2611.23879v1 Announce Type: new Abstract: Stateful Serverless (Function-as-a-Service) environment are increasingly used for stream processing applications. These applications are often structured as pipelines where data flows through a sequence of distinct processing operators. Many operators are stateful and require access to the processing state during execution. To address this, stateful serverless platforms introduce state servers on worker nodes, co-locating state with computation to dramatically reduce access latency. In serverless, these data pipelines are realized by chaining function invocations, passing the processed data from one function to the next. To reduce network overhead, state-of-the-art scheduling approaches employ a distributed worker-level scheduling model, routing locally generated chained requests directly to destination workers, bypassing centralized schedulers. However, these approaches always co-locate state and execution for the same operator in the same worker, which forces most invocations to the next operator to be transferred across workers, thereby increasing both latency and I/O overhead. As a result, this paper proposes LcSched, a novel distributed scheduling strategy tailored for stream processing applications in stateful serverless environments. LcSched seeks to maximise local execution of chained function invocations while ensuring that each execution accesses its state from the local worker. It also monitors workload at runtime to maintain workload balance between workers. Experimental results show that LcSched improves throughput by up to 3.6x and reduces end-to-end latency by up to 79% compared to the state-of-the-art distributed scheduling strategy for stateful serverless environments.

## A modern dilemma: Consume data from a stream or a lake

DevFeed: [A modern dilemma: Consume data from a stream or a lake](<https://devfeed.tech/articles/a-modern-dilemma-consume-data-from-a-stream-or-a-lake-141161.md>)

Original publisher: [Read original article](<https://medium.com/capital-one-tech/a-modern-dilemma-consume-data-from-a-stream-or-a-lake-296bbb354094?source=rss----3db3a67cb648---4>)

Author: Capital One Tech

Published: 2026-11-18T17:35:14Z

Content type: article

Language: en

Sources: [Capital One Tech](<https://devfeed.tech/sources/capital-one-tech.md>)

Topics: [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [etl](<https://devfeed.tech/topics/etl.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [apis](<https://devfeed.tech/tags/apis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [batch](<https://devfeed.tech/tags/batch.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [business-processes](<https://devfeed.tech/tags/business-processes.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [enrichment](<https://devfeed.tech/tags/enrichment.md>), [etl](<https://devfeed.tech/tags/etl.md>), [event](<https://devfeed.tech/tags/event.md>), [flink](<https://devfeed.tech/tags/flink.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [latency](<https://devfeed.tech/tags/latency.md>), [legacy](<https://devfeed.tech/tags/legacy.md>), [log](<https://devfeed.tech/tags/log.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [model-context-protocol](<https://devfeed.tech/tags/model-context-protocol.md>), [software-architecture](<https://devfeed.tech/tags/software-architecture.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [stream](<https://devfeed.tech/tags/stream.md>)

### AI overview

Choosing a data consumption pattern early can avoid costly architectural changes later. APIs support synchronous, record-by-record requests and paginated multirecord results, making them suitable for low-latency needs; Model Context Protocol can provide similar access for agents and is often an API wrapper. Event streams, commonly built with Kafka, Flink or Spark Streaming, suit sub-second latency and enable per-event calculations, transformations and enrichment. The excerpt also identifies Delta Change Data Feed (CDF) as a record-by-record pattern and traditional batch ETL as a multirecord pattern, but ends before explaining CDF's mechanics or comparing these approaches further.

### Source excerpt

Data is a critical component of software architecture, providing the foundation for business processes, including those powered by artificial intelligence (AI). Quality, volume, latency, consistency, ordering and duplicates are essential considerations in software design. Modern cloud architectures consume data in various ways, such as through application programming interfaces (APIs), streams and data lakes. You can consume data at the individual event or record level as it arrives or in larger batches. Legacy architectures often process data in scheduled batches, forcing surrounding architecture and business processes to depend on that approach. Getting the data consumption approach right up front is vital, because rearchitecting later is costly. General patterns to consume data There are several general data consumption patterns. These include APIs (record-by-record or multirecord results), event stream consumption (record-by-record), Delta Change Data Feed (CDF) consumption (record-by-record) or traditional batch ETL-extract, transform, load (multirecord results). APIs are ideal for low-latency, record-by-record requirements. They also support multirecord results, typically through pagination. APIs usually follow a synchronous approach, where the client makes a call and waits for a response before moving to the next step. Model Context Protocol is an industry specification for agents to consume data and can support record-by-record or batch results via pagination, similar to APIs. They are often a wrapper on top of APIs. The event stream pattern works well for sub-second latency requirements and typically runs on a tech stack like Kafka, Flink or Spark Streaming. These technologies excel at record-level calculations (like on-the-fly calculation), transformations (like column mapping) or enrichment (like adding data to an existing event). Delta Lake is often used in data lakes and serves as a transaction log to standard Apache Parquet data files (source). CDF is

## A Temporal Algebraic Framework for Data Stream Evaluation Under Delayed Labels and Distribution Shifts

DevFeed: [A Temporal Algebraic Framework for Data Stream Evaluation Under Delayed Labels and Distribution Shifts](<https://devfeed.tech/articles/a-temporal-algebraic-framework-for-data-stream-evaluation-under-delayed-labels-and-distribution-shifts-139926.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21914>)

Author: Ronaldo C. Prati, Elaine Ribeiro de Faria

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Lean](<https://devfeed.tech/topics/lean.md>)

Tags: [computer-science](<https://devfeed.tech/tags/computer-science.md>), [concept-drift](<https://devfeed.tech/tags/concept-drift.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [lean-4](<https://devfeed.tech/tags/lean-4.md>), [mask](<https://devfeed.tech/tags/mask.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [package](<https://devfeed.tech/tags/package.md>), [precision](<https://devfeed.tech/tags/precision.md>), [recall](<https://devfeed.tech/tags/recall.md>), [sliding-window](<https://devfeed.tech/tags/sliding-window.md>), [specificity](<https://devfeed.tech/tags/specificity.md>), [stream](<https://devfeed.tech/tags/stream.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [temporal](<https://devfeed.tech/tags/temporal.md>)

### AI overview

The framework extends confusion-matrix-based micro and macro evaluation to data streams using exponentially faded or sliding-window counts, with O(m²) updates per step. It handles delayed or selectively observed labels by temporarily counting instances through calibrated class probabilities; interim bias depends on calibration error, and classes without verified instances receive no value under audited conventions. To distinguish prevalence shifts from concept drift, it monitors recall and specificity and uses a first-order Jacobian decomposition to attribute metric changes. On synthetic streams and four real datasets, a detector using these prior-free coordinates and per-class memory produced no excess false alarms under label shift, unlike the Jacobian alarm and error-rate detectors. Delayed-label correction recovered recall in some real settings but only partially corrected precision, specificity, and F1. The abstract reports 25 statements checked in Lean 4 and code in the aggmetrics package.

### Source excerpt

arXiv:2611.21914v1 Announce Type: cross Abstract: In data streams, labels often arrive late or only for some instances, class prevalence drifts, and rare classes vanish from short windows. Standard evaluation then confuses a degraded decision boundary with a change in prevalence, and is biased when labels are delayed or selected by the model. We extend an algebraic framework that obtains pooled (micro) and per-class (macro) measures from one confusion matrix to streams, weighting the counts in time by exponential fading or a sliding window; the update costs O(m^2) per step, whatever the stream length, and each pooled measure stays a weighted combination of the per-class ones. Delayed and selective feedback enter as an observation mask: until its label arrives, an instance is counted with the model's calibrated class probabilities, and the bias of this interim estimate is driven by their calibration error. Audited conventions report no value for a class without verified instances. We separate prior shift from concept drift by monitoring prior-free coordinates (recall and specificity) and attribute a change in a measure to the two sources with a first-order Jacobian decomposition. On synthetic streams and four real datasets, a detector on these coordinates with a memory per class raises no excess false alarms under label shift, unlike the Jacobian alarm and error-rate detectors. The correction for delayed labels recovers recall in some real settings and is only partial for precision, specificity and F1. Twenty-five statements are checked in Lean 4, and the code is in the aggmetrics package.

## Reduce the complexity and cost of real-time processing with queues in Amazon Kinesis Data Streams

DevFeed: [Reduce the complexity and cost of real-time processing with queues in Amazon Kinesis Data Streams](<https://devfeed.tech/articles/reduce-the-complexity-and-cost-of-real-time-processing-with-queues-in-amazon-kinesis-data-streams-140167.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/reduce-the-complexity-and-cost-of-real-time-processing-with-queues-in-amazon-kinesis-data-streams/>)

Author: Dirk Fröhner

Published: 2026-11-17T23:17:16Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-kinesis](<https://devfeed.tech/tags/amazon-kinesis.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [backlog](<https://devfeed.tech/tags/backlog.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [demand](<https://devfeed.tech/tags/demand.md>), [instead](<https://devfeed.tech/tags/instead.md>), [kinesis](<https://devfeed.tech/tags/kinesis.md>), [kinesis-data-streams](<https://devfeed.tech/tags/kinesis-data-streams.md>), [offers](<https://devfeed.tech/tags/offers.md>), [one](<https://devfeed.tech/tags/one.md>), [processing](<https://devfeed.tech/tags/processing.md>), [queues](<https://devfeed.tech/tags/queues.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [streams](<https://devfeed.tech/tags/streams.md>), [time](<https://devfeed.tech/tags/time.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Amazon Kinesis Data Streams queues provide per-record delivery tracking for applications that can process records independently. Consumers receive batches of up to 10 records, process them, and acknowledge them; unacknowledged records become available again after a configurable visibility timeout. Delivery is at least once and ordering is best effort, so processing should be idempotent. Records that exceed the default six receives without acknowledgment go to a dead-letter state, where operators can inspect, restore, or discard them. Queues scale consumer instances with backlog rather than shard count and remove the need for a KCL lease table and per-shard custom metrics. Multiple queues, stream consumers, and managed delivery can use the same stream without a relay or duplicate ingest. Use direct stream consumers when processing depends on record order or prior state; use queues for independent work. AWS reports up to 43% lower processing cost, but its synthetic benchmark used an I/O-bound handler; savings depend on workload and configuration, and CPU-bound work narrows the compute benefit.

### Source excerpt

Amazon Kinesis Data Streams now offers queues, a new way to consume the records of an on-demand stream. A queue delivers records one by one, tracks each record's delivery state, redelivers unacknowledged records, sets persistently failing records aside as dead letters, and scales with your backlog instead of your shard count.

## Random Cut Forests, Games and Diffusion over Approximately Nested Anisotropic Local Geometry

DevFeed: [Random Cut Forests, Games and Diffusion over Approximately Nested Anisotropic Local Geometry](<https://devfeed.tech/articles/random-cut-forests-games-and-diffusion-over-approximately-nested-anisotropic-local-geometry-138885.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19548>)

Author: Sudipto Guha

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [TensorFlow Decision Forests](<https://devfeed.tech/topics/tensorflow-decision-forests.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [added](<https://devfeed.tech/tags/added.md>), [affinity](<https://devfeed.tech/tags/affinity.md>), [align](<https://devfeed.tech/tags/align.md>), [also](<https://devfeed.tech/tags/also.md>), [ambient](<https://devfeed.tech/tags/ambient.md>), [approach](<https://devfeed.tech/tags/approach.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [axis](<https://devfeed.tech/tags/axis.md>), [based](<https://devfeed.tech/tags/based.md>), [basis](<https://devfeed.tech/tags/basis.md>), [but](<https://devfeed.tech/tags/but.md>), [calculus](<https://devfeed.tech/tags/calculus.md>), [change](<https://devfeed.tech/tags/change.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [games](<https://devfeed.tech/tags/games.md>), [local](<https://devfeed.tech/tags/local.md>), [nested](<https://devfeed.tech/tags/nested.md>), [over](<https://devfeed.tech/tags/over.md>), [random](<https://devfeed.tech/tags/random.md>)

### AI overview

The paper uses the evolving geometry of random cut forests as a shared energy framework for streaming prediction, explanation, games, and diffusion. Existing scores are interpreted as random-walk escape probabilities, and the resulting affinity is represented exactly by a sparse electrical network on each tree, maintained along one path per update. Different clamp choices yield harmonic predictions with Gaussian uncertainty, normalized cuts with costs attributed to oriented faces, or coalition games that measure field response. With noise, the energy also defines diffusion dynamics whose relaxation rates come from the boxes. The authors state that localization of axis-parallel cuts is controlled by affine rather than ambient dimension; across a forest, boxes are only approximately nested and are aligned using common reference observations. A noisy rotating-star example illustrates the measurements.

### Source excerpt

arXiv:2611.19548v1 Announce Type: new Abstract: Just as the data in a dynamic stream changes, questions asked of the stream also change in intent: what to hold constant, which counterfactuals to consider, and so forth. Energy-based learning has been celebrated for its uniform approach to losses and estimation, but interpretability remains challenging. Tree-based methods, on the other hand, offer high interpretability but often rely on bespoke, disjointed estimators. Neither paradigm readily supports changing questions on streaming data. In this paper we take a step towards combining energy, interpretability, and streaming. Random cut forests retain an evolving geometry of the data. We study this geometry as the basis for diffusion dynamics, prediction, explanation, and games. We show that several existing scores are escape probabilities of random walks on the trees. The induced affinity is realized exactly by a sparse electrical network on each tree, giving a single energy calculus, a Hamiltonian, maintained along one path per update. This connects an estimate, its supporting evidence, and the response to intervention through the same boxes, gaps, masses, and paths. Different questions correspond to different clamp decisions on the energy. Fixing observed values gives harmonic prediction with Gaussian uncertainty. Fixing a partition gives normalized cuts whose cost is attributed exactly to oriented faces. Releasing clamps gives coalition games, in which collusive displacement measures the field response. With noise, the same energy gives diffusion dynamics whose relaxation rates are read from the boxes. Axis-parallel cuts localize at a rate set by affine rather than ambient dimension, and this control passes to the averaged energy. Across a forest the boxes are only approximately nested; common reference observations align the trees. A rotating star with added noise illustrates the measurements.

## Rate-Limiting Iceberg Structured Streaming: Micro-Batches, Async Planning, and Snapshot Safety

DevFeed: [Rate-Limiting Iceberg Structured Streaming: Micro-Batches, Async Planning, and Snapshot Safety](<https://devfeed.tech/articles/rate-limiting-iceberg-structured-streaming-micro-batches-async-planning-and-snapshot-safety-123567.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/rate-limiting-iceberg-structured-streaming-micro-batches-async-planning-and-snapshot-safety/>)

Author: Alex Merced

Published: 2026-11-09T13:41:56Z

Content type: article

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [async](<https://devfeed.tech/tags/async.md>), [batch](<https://devfeed.tech/tags/batch.md>), [checkpoint](<https://devfeed.tech/tags/checkpoint.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [file](<https://devfeed.tech/tags/file.md>), [handling](<https://devfeed.tech/tags/handling.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [limits](<https://devfeed.tech/tags/limits.md>), [planning](<https://devfeed.tech/tags/planning.md>), [rate-limiting](<https://devfeed.tech/tags/rate-limiting.md>), [safety](<https://devfeed.tech/tags/safety.md>), [snapshot](<https://devfeed.tech/tags/snapshot.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [structured](<https://devfeed.tech/tags/structured.md>)

### AI overview

Spark's Iceberg streaming source can bound micro-batch work with a maximum file count and an estimated row limit. The row limit uses file metadata and does not split files, so a large file may exceed it; tune limits using per-file statistics. Asynchronous planning moves manifest expansion to background work so executors can start sooner, but it does not alter file selection or snapshot semantics and can still raise driver memory use. Operators must choose how to handle append, overwrite, and delete snapshots, keep checkpoints durable, and test restarts and checkpoint identity changes, which can replay or skip work. The article recommends staging tests, conservative limits, and monitoring planning time, manifest counts, driver memory, batch duration, and sink duplicates or gaps. Verify option names and behavior against the Iceberg and Spark versions in use; exact row slicing requires smaller input files or another ingestion pattern.

### Source excerpt

Control Iceberg Spark Structured Streaming with micro-batch file and row limits, async planning, checkpoint safety, and overwrite handling. The post Rate-Limiting Iceberg Structured Streaming: Micro-Batches, Async Planning, and Snapshot Safety appeared first on Dremio.

## Load Balancing Starts Before the Load Balancer: Routing Patterns for Distributed Systems

DevFeed: [Load Balancing Starts Before the Load Balancer: Routing Patterns for Distributed Systems](<https://devfeed.tech/articles/load-balancing-starts-before-the-load-balancer-routing-patterns-for-distributed-systems-121760.md>)

Original publisher: [Read original article](<https://hackernoon.com/load-balancing-starts-before-the-load-balancer-routing-patterns-for-distributed-systems?source=rss>)

Author: Varsha Ganesh

Published: 2026-11-06T22:19:26Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>), [consistent hashing](<https://devfeed.tech/topics/consistent-hashing.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Memcached](<https://devfeed.tech/topics/memcached.md>), [nginx](<https://devfeed.tech/topics/nginx.md>), [IP routing](<https://devfeed.tech/topics/ip-routing.md>)

Tags: [cache](<https://devfeed.tech/tags/cache.md>), [capacity-planning](<https://devfeed.tech/tags/capacity-planning.md>), [cassandra](<https://devfeed.tech/tags/cassandra.md>), [consistent-hashing](<https://devfeed.tech/tags/consistent-hashing.md>), [control-plane](<https://devfeed.tech/tags/control-plane.md>), [databases](<https://devfeed.tech/tags/databases.md>), [distributed-database](<https://devfeed.tech/tags/distributed-database.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [load-balancers](<https://devfeed.tech/tags/load-balancers.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [partitions](<https://devfeed.tech/tags/partitions.md>), [routing](<https://devfeed.tech/tags/routing.md>), [routing-patterns](<https://devfeed.tech/tags/routing-patterns.md>), [stateless-services](<https://devfeed.tech/tags/stateless-services.md>), [stream](<https://devfeed.tech/tags/stream.md>), [streams](<https://devfeed.tech/tags/streams.md>)

### AI overview

For stateless services, DNS or a proxy can distribute requests among healthy interchangeable backends; DNS is coarse and health probes add cost without showing relative load. Stateful systems require routing tied to data placement, so control planes must distribute partitions across hosts while honoring replication, failure domains, and hardware capacity. That placement can still produce hot partitions when particular keys attract disproportionate traffic. The article recommends partition-level metrics, thresholds that account for normal variance, and key attribution, followed by incremental, reversible live splits with routing metadata updated during the transition. Splitting will not help if a single hot key remains the cause. For caches, consistent hashing eases shard rebalancing but not hot keys; request coalescing and local caches can absorb spikes. For Kafka or Flink, ordering ties each key to a partition, making runtime splitting constrained: partition count and key distribution require advance planning. More partitions enable consumer parallelism but increase broker recovery, rebalance, and efficiency costs, and do not solve a single hot key.

### Source excerpt

Load balancing isn't just a load balancer. Learn how routing works for databases, caches, and stream processors, and how to handle hot partitions

## Stream your customers' audit logs into their own Snowflake warehouse

DevFeed: [Stream your customers' audit logs into their own Snowflake warehouse](<https://devfeed.tech/articles/stream-your-customers-audit-logs-into-their-own-snowflake-warehouse-121079.md>)

Original publisher: [Read original article](<https://workos.com/blog/audit-log-streaming-snowflake>)

Author: WorkOS

Published: 2026-11-05T00:00:00Z

Content type: tutorial

Language: en

Sources: [WorkOS](<https://devfeed.tech/sources/workos-blog.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [audit](<https://devfeed.tech/topics/audit.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2026](<https://devfeed.tech/tags/2026.md>), [3](<https://devfeed.tech/tags/3.md>), [audit](<https://devfeed.tech/tags/audit.md>), [audit-logs](<https://devfeed.tech/tags/audit-logs.md>), [auth](<https://devfeed.tech/tags/auth.md>), [customer](<https://devfeed.tech/tags/customer.md>), [customers](<https://devfeed.tech/tags/customers.md>), [event](<https://devfeed.tech/tags/event.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [key](<https://devfeed.tech/tags/key.md>), [logs](<https://devfeed.tech/tags/logs.md>), [pair](<https://devfeed.tech/tags/pair.md>), [schema](<https://devfeed.tech/tags/schema.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [stream](<https://devfeed.tech/tags/stream.md>), [table](<https://devfeed.tech/tags/table.md>), [up](<https://devfeed.tech/tags/up.md>), [warehouse](<https://devfeed.tech/tags/warehouse.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

WorkOS Log Streams can write each organization's audit events as rows in the customer's own Snowflake account, where the customer controls storage and retention. Setup requires a dedicated user authenticated by an RSA key pair, a table with the documented schema, and a default role granted access to the warehouse, database, schema, and table; accounts with network policies must allow WorkOS's published IP addresses. The dashboard and Admin Portal configure the same stream, but the portal lets the customer provide credentials directly so the private key need not pass through support. Monitor actual delivery with a customer-specific freshness query in Snowflake: an Active state alone does not establish that rows are arriving, and the article says not to assume stop-delivery notifications. Error or Invalid streams require fixing the cause and revalidating. Delivery resumes from the last successful event, subject to the organization's WorkOS retention window; previously delivered rows remain subject to the customer's own warehouse policy.

### Source excerpt

Each audit event lands as a row in your customer's own Snowflake table, with key-pair auth, a fixed schema, grants up front, and a freshness check.

## Mastering stateful stream reordering at 200,000 messages per second

DevFeed: [Mastering stateful stream reordering at 200,000 messages per second](<https://devfeed.tech/articles/mastering-stateful-stream-reordering-at-200-000-messages-per-second-117179.md>)

Original publisher: [Read original article](<https://blogs.cisco.com/cisco-devnet/flink-ab-development-stateful-stream-processing>)

Author: Soumitra Kumar

Published: 2026-11-04T17:00:06Z

Content type: tutorial

Language: en

Sources: [Cisco Blogs](<https://devfeed.tech/sources/cisco-blogs.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Stateful](<https://devfeed.tech/topics/stateful.md>), [rocksdb](<https://devfeed.tech/topics/rocksdb.md>)

Tags: [amazon-eks](<https://devfeed.tech/tags/amazon-eks.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [buffer](<https://devfeed.tech/tags/buffer.md>), [checkpoint](<https://devfeed.tech/tags/checkpoint.md>), [cisco-devnet](<https://devfeed.tech/tags/cisco-devnet.md>), [devnet](<https://devfeed.tech/tags/devnet.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [exactly-once](<https://devfeed.tech/tags/exactly-once.md>), [flink](<https://devfeed.tech/tags/flink.md>), [memory](<https://devfeed.tech/tags/memory.md>), [netops](<https://devfeed.tech/tags/netops.md>), [rocksdb](<https://devfeed.tech/tags/rocksdb.md>), [stateful](<https://devfeed.tech/tags/stateful.md>), [stream](<https://devfeed.tech/tags/stream.md>)

### AI overview

At 200,000 messages per second, per-event RocksDB reads and writes plus timer registration slowed the Flink job and contributed to false positives in connection-success reporting when events arrived late or out of order. The replacement buffers each session's events as JVM heap objects in a KeyedProcessFunction, sorts by event time, and snapshots the buffer to Flink ListState at checkpoints for recovery. This removes constant state-backend access from the hot path, with the trade-off that heap use must be bounded. The team used three-minute idle eviction, timer debouncing, and compact records, then ran the new variant beside the original on production data for five months before switching default topics. The source notes that end-to-end exactly-once behavior also depends on sink handling of replayed output.

### Source excerpt

Our Flink job hit a wall at 200,000 messages per second because of RocksDB. Here is the in-memory reordering pattern that fixed it, and how A/B development proved it safe.

## Your Business, Always Current. Streamhouse - The Data Architecture for the Age of AI.

DevFeed: [Your Business, Always Current. Streamhouse - The Data Architecture for the Age of AI.](<https://devfeed.tech/articles/your-business-always-current-streamhouse-the-data-architecture-for-the-age-of-ai-117675.md>)

Original publisher: [Read original article](<https://www.confluent.io/blog/streamhouse-your-business-always-current/>)

Author: Yashwanth Dasari

Published: 2026-11-04T15:52:47Z

Content type: article

Language: en

Sources: [Confluent: Data in motion](<https://devfeed.tech/sources/confluent-data-in-motion.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [data-governance](<https://devfeed.tech/topics/data-governance.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [business logic](<https://devfeed.tech/topics/business-logic.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [about](<https://devfeed.tech/tags/about.md>), [account](<https://devfeed.tech/tags/account.md>), [age-of-ai](<https://devfeed.tech/tags/age-of-ai.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [applications](<https://devfeed.tech/tags/applications.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [business](<https://devfeed.tech/tags/business.md>), [confluent-cloud](<https://devfeed.tech/tags/confluent-cloud.md>), [context](<https://devfeed.tech/tags/context.md>), [data](<https://devfeed.tech/tags/data.md>), [data-architecture](<https://devfeed.tech/tags/data-architecture.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [making](<https://devfeed.tech/tags/making.md>), [processes](<https://devfeed.tech/tags/processes.md>), [state](<https://devfeed.tech/tags/state.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

A Streamhouse is presented as an architecture for keeping reusable business state and context current across distributed systems so applications, AI agents, and analytics can act on it. It combines historical data with ongoing operational changes, maintains state incrementally, and supports replay, correction, and recomputation. Its three requirements are freshness with history, decentralized sharing without consolidating onto one platform, and mission-critical correctness, availability, security, recovery, and traceability. The described components include connectors, a durable event backbone, stateful processing, shared semantics and governance, and serving through streams, queries, APIs, or tables. Operational systems still authorize transactions; warehouses and lakehouses still support historical analysis and training. The architecture does not guarantee a globally synchronized "now," and implementations trade off latency, consistency, replay, cost, and operational complexity. The text gives examples including Siemens Healthineers, Neuron Systems, and Yelp, and presents Confluent's platform as one implementation; its claims about platform completeness are vendor claims.

### Source excerpt

A Streamhouse is a data architecture that continuously captures, processes, governs and serves current business state and context across distributed systems, making them available to applications, AI agents, and analytics.

## StreamLTT: Risk Certification for Policy Selection under Stream Dependence

DevFeed: [StreamLTT: Risk Certification for Policy Selection under Stream Dependence](<https://devfeed.tech/articles/streamltt-risk-certification-for-policy-selection-under-stream-dependence-116887.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.04062>)

Author: Jinwen Xu, Qin Lu

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [block](<https://devfeed.tech/tags/block.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [budget](<https://devfeed.tech/tags/budget.md>), [centering](<https://devfeed.tech/tags/centering.md>), [certification](<https://devfeed.tech/tags/certification.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [error](<https://devfeed.tech/tags/error.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [math-st](<https://devfeed.tech/tags/math-st.md>), [policy](<https://devfeed.tech/tags/policy.md>), [region](<https://devfeed.tech/tags/region.md>), [risk](<https://devfeed.tech/tags/risk.md>), [risks](<https://devfeed.tech/tags/risks.md>), [selection](<https://devfeed.tech/tags/selection.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [stat-th](<https://devfeed.tech/tags/stat-th.md>), [stream](<https://devfeed.tech/tags/stream.md>)

### AI overview

StreamLTT certifies policy risk for dependent, potentially nonstationary deployment streams by targeting average predictable risk: each loss is conditioned on information available before its outcome. Conditional centering makes deviations a martingale sum, supporting a predictable-variance-adaptive test and a finite-sample Hoeffding alternative. With family-wise error control, the method produces a simultaneously certified policy region for later selection. Across seven forecasting datasets and seven deployment risks, the reported certified frontier was closer to the target than IID, robust-variance, block-bootstrap, and concentration-based alternatives, while empirical family-wise error tracked its prescribed level. An ablation attributes the tighter frontier to predictable-risk formulation and more effective use of the error budget to variance adaptation.

### Source excerpt

arXiv:2611.04062v1 Announce Type: new Abstract: Learn-then-Test (LTT) provides a general framework for certifying deployment policies against a prescribed risk target, but its standard formulation targets a fixed population risk estimated from calibration losses. In sequential deployment, losses may instead be dependent and nonstationary, so the risk of a fixed policy can evolve with the observed history. We introduce StreamLTT, which replaces this fixed population risk with average predictable risk, defined by conditioning each decision's loss on the information available before its outcome is revealed. Conditional centering turns empirical deviations from this target into a martingale sum, enabling both a test that adapts to predictable variance and a finite-sample Hoeffding alternative. Combined with family-wise error rate control, StreamLTT yields a simultaneously certified policy region that supports subsequent deployment selection. Across seven forecasting datasets and seven deployment risks, StreamLTT places the certified frontier substantially closer to the prescribed risk target than IID, robust-variance, block-bootstrap, and concentration-based alternatives, while empirical family-wise error closely follows the prescribed level. A finite-sample ablation further shows that the predictable-risk formulation drives the tighter risk frontier, while adaptation to predictable variance makes more effective use of the available family-wise error budget. StreamLTT therefore extends risk-certified policy selection to dependent deployment streams by addressing both what risk should be certified and how it should be tested.

## Online Label-Free Gradient Matching for Continual Language Model Training

DevFeed: [Online Label-Free Gradient Matching for Continual Language Model Training](<https://devfeed.tech/articles/online-label-free-gradient-matching-for-continual-language-model-training-115453.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.03496>)

Author: Sajjad Pakdamansavoji, Mohammad Reza Rezaei, Ilija Subasic, Pedro Garzon, Radu Paradovschi, Zhao Chen

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [data](<https://devfeed.tech/tags/data.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [online](<https://devfeed.tech/tags/online.md>), [text-classification](<https://devfeed.tech/tags/text-classification.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

OLGA creates compact training sets for continual language-model updates by matching gradients against targets from the model's own next-token predictions, so it can distill unlabeled data. It maintains a rehearsal memory of synthetic examples and re-distills that memory with each incoming batch. Across five text-classification benchmarks using phi-1.5, it improved accuracy by 10 percentage points over the strongest selection baseline with as few as five examples; online, it exceeded the strongest selection-based continual baseline by 6.6 stream-AUC points. A hybrid pipeline uses selection for mild compression and distillation for extreme compression, reporting 30% lower runtime than distillation alone at similar accuracy and up to 12.6 percentage points better results than the best selection-only pipeline.

### Source excerpt

arXiv:2611.03496v1 Announce Type: new Abstract: Language models in customer-facing products must be continually updated, since the data they serve shifts over time. As retraining on the full accumulated data is impractical, each update should instead run on a small set of examples carrying the training signal of all data seen so far. The strongest current methods build this set by gradient matching, which synthesizes examples whose training gradients match those of the full data. However, they require ground-truth labels, run once on a fixed snapshot rather than on a growing stream, and are slow on large data. We introduce \textbf{OLGA}, which lifts these limitations. It distills unlabeled data by drawing the gradient matching targets from the model's own next-token predictions. Across five text classification benchmarks with phi-1.5, it improves accuracy by 10 percentage points over the strongest selection baseline using as few as five examples. OLGA also distills online data streams by keeping a running rehearsal memory of synthetic examples, which is re-distilled together with each new batch as it arrives. In the online setting, it outperforms the strongest selection-based continual baseline by 6.6 stream-AUC percentage points. Finally, our method scales to large data by pairing distillation with selection in a hybrid pipeline, letting selection handle the mild stage of compression and distillation the extreme one. It cuts runtime by 30\% versus distillation alone at similar accuracy, and beats the best selection-only pipeline by up to 12.6 percentage points.

## Live Adaptation of Dataflow Stream Processing for Hybrid Edge-Cloud Analytics

DevFeed: [Live Adaptation of Dataflow Stream Processing for Hybrid Edge-Cloud Analytics](<https://devfeed.tech/articles/live-adaptation-of-dataflow-stream-processing-for-hybrid-edge-cloud-analytics-101622.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.34658>)

Author: Yuanli Wang, Prateek Jain, Lei Huang, Le Xu, Vasiliki Kalavri

Published: 2026-10-27T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [applications](<https://devfeed.tech/tags/applications.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [business-analytics](<https://devfeed.tech/tags/business-analytics.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [data](<https://devfeed.tech/tags/data.md>), [datacenter](<https://devfeed.tech/tags/datacenter.md>), [dataflow](<https://devfeed.tech/tags/dataflow.md>), [devices](<https://devfeed.tech/tags/devices.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [edge](<https://devfeed.tech/tags/edge.md>), [execution](<https://devfeed.tech/tags/execution.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [live](<https://devfeed.tech/tags/live.md>), [processing](<https://devfeed.tech/tags/processing.md>), [stream](<https://devfeed.tech/tags/stream.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>)

### AI overview

Sluice adapts streaming computations between edge devices and cloud resources as network and edge conditions change, without pausing execution. Its automatic query rewriter divides applications into independent fault-tolerant segments; live routing shifts their computations while preserving event-time semantics. The system is designed to work with existing dataflow engines and is implemented on Apache Flink. Evaluation across five real-world applications used Raspberry Pi devices and a simulated 100-VM testbed; the abstract reports higher throughput than baseline systems, but gives no numerical results or specific baseline details.

### Source excerpt

arXiv:2610.34658v1 Announce Type: new Abstract: Dataflow streaming systems, like Apache Flink and Spark Streaming, are integral to modern business analytics, providing real-time insights across industries. However, for emerging hybrid edge-cloud analytics pipelines, neither processing everything in the cloud nor static edge offloading can reliably sustain the target rate under dynamic WAN conditions and limited edge resources. In this paper, we explore how these datacenter-centric platforms can be extended to support such hybrid analytics pipelines. Our goal is to maintain full transparency for user applications, while retaining built-in fault tolerance, out-of-order processing, and existing ecosystem integrations. Achieving this is challenging because existing dataflow engines assume homogeneous resources, rely on static task placement, and do not support adaptation under dynamic network and edge conditions. We introduce Sluice, an adaptive data stream processing system for hybrid clouds, compatible with dataflow streaming engines. Sluice's key contributions are: (i) an automatic query rewriter that decomposes applications into independent, fault-tolerant segments, and (ii) a live routing mechanism that shifts computations between edge and cloud without pausing execution while preserving event-time semantics. We implement Sluice on Apache Flink and evaluate it on a physical testbed with Raspberry Pi devices, as well as on a large simulated testbed of 100 VMs. Across five real-world applications, our results show that Sluice seamlessly shifts computations between edge and cloud, while consistently achieving higher throughput than baseline systems.

## New Forrester study: Why modernizing data streaming is about more than cost savings

DevFeed: [New Forrester study: Why modernizing data streaming is about more than cost savings](<https://devfeed.tech/articles/new-forrester-study-why-modernizing-data-streaming-is-about-more-than-cost-savings-101015.md>)

Original publisher: [Read original article](<https://www.confluent.io/blog/new-forrester-study-why-modernizing-data-streaming-is-about-more-than-cost-savings/>)

Author: Mike Agnich

Published: 2026-10-26T17:33:56Z

Content type: opinion

Language: en

Sources: [Confluent: Data in motion](<https://devfeed.tech/sources/confluent-data-in-motion.md>)

Topics: [Streaming](<https://devfeed.tech/topics/streaming.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [it modernization](<https://devfeed.tech/topics/it-modernization.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [about](<https://devfeed.tech/tags/about.md>), [access-controls](<https://devfeed.tech/tags/access-controls.md>), [ai](<https://devfeed.tech/tags/ai.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [application](<https://devfeed.tech/tags/application.md>), [attention](<https://devfeed.tech/tags/attention.md>), [based](<https://devfeed.tech/tags/based.md>), [benefit](<https://devfeed.tech/tags/benefit.md>), [best](<https://devfeed.tech/tags/best.md>), [confluent](<https://devfeed.tech/tags/confluent.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cost-savings](<https://devfeed.tech/tags/cost-savings.md>), [data](<https://devfeed.tech/tags/data.md>), [data-streaming](<https://devfeed.tech/tags/data-streaming.md>), [forrester](<https://devfeed.tech/tags/forrester.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [study](<https://devfeed.tech/tags/study.md>)

### AI overview

The commissioned Forrester TEI study attributes most of the composite organization's quantified benefit to engineering capacity recovered after moving from self-managed open-source Kafka to Confluent Cloud. Annual platform operations and enablement requirements fell from 10.5 FTEs to 3.5, accounting for USD 2.84 million of USD 3.58 million in three-year benefits; incidents declined from 12 to 4 per year, and each new or materially updated production streaming use case avoided 360 technical delivery hours. The modeled organization achieved 137% ROI, USD 2.1 million NPV and a 10-month payback. Interviewees also reported reuse, faster onboarding and more consistent governance; one described loyalty points posting in 15 minutes instead of two days. These figures come from a three-year composite modeled from four customer interviews and are not necessarily representative of all customers.

### Source excerpt

Confluent blog

## Evolving Federated Density Peak Clustering with Shuffler-Based Anonymization

DevFeed: [Evolving Federated Density Peak Clustering with Shuffler-Based Anonymization](<https://devfeed.tech/articles/evolving-federated-density-peak-clustering-with-shuffler-based-anonymization-96605.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.31211>)

Author: Miha O\\v{z}bot, Seiichi Ozawa, Igor \\v{S}krjanc

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [clustering](<https://devfeed.tech/topics/clustering.md>), [Federated Learning](<https://devfeed.tech/topics/federated-learning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [clustering](<https://devfeed.tech/tags/clustering.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-sy](<https://devfeed.tech/tags/cs-sy.md>), [data-privacy](<https://devfeed.tech/tags/data-privacy.md>), [eess-sy](<https://devfeed.tech/tags/eess-sy.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [federated](<https://devfeed.tech/tags/federated.md>), [federated-learning](<https://devfeed.tech/tags/federated-learning.md>), [local](<https://devfeed.tech/tags/local.md>), [private](<https://devfeed.tech/tags/private.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [stream](<https://devfeed.tech/tags/stream.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>)

### AI overview

eFDPC is a one-shot federated clustering method that reduces client-server exchanges while anonymizing client contributions. Each client builds evolving micro-clusters from private data mini-batches by adding and merging clusters, then sends only density peaks instead of its full local model. A shuffler anonymizes those contributions before the server reconstructs them and selects global peaks. The paper describes eFDPC-P, which sends peaks only, and eFDPC-G, which adds server-side label propagation. Experiments on synthetic and real-world benchmarks report that eFDPC outperforms recent federated clustering baselines in one-shot settings and scales to higher-dimensional, non-IID data.

### Source excerpt

arXiv:2610.31211v1 Announce Type: cross Abstract: In this paper, we present a novel one-shot federated clustering approach based on evolving fuzzy clustering and shuffler-based anonymization. In federated clustering, each round reveals additional model information to the server and can increase the opportunity to infer properties of local datasets, even when raw samples are never shared. Consequently, fewer client-server exchanges and unlinkability are important, so transmitted information cannot be attributed to any single client. Data-stream clustering algorithms, which learn in a single pass over the data, align well with the one-shot federated learning paradigm, where a single communication round can be sufficient. The proposed Evolving Federated Density Peak Clustering (eFDPC) method combines evolving fuzzy data-stream clustering and density peak clustering in a federated setting. Each client independently evolves micro-clusters from private data mini-batches through cluster addition and merging. To limit communication, density peak clustering is applied to the learned micro-clusters and only the selected peaks are transmitted, rather than the full local models. A shuffler-based layer is introduced between the clients and the server to anonymize contributions prior to reconstruction and global peak selection. We consider a peak-only variant, eFDPC-P, as well as eFDPC-G with server-side label propagation. Experiments on synthetic and real-world benchmark datasets show that eFDPC outperforms recent federated clustering baselines in one-shot settings and scales to higher-dimensional, non-IID data.

## Fault Tolerance for Stream Processing Engines

DevFeed: [Fault Tolerance for Stream Processing Engines](<https://devfeed.tech/articles/fault-tolerance-for-stream-processing-engines-93729.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/1605.00928>)

Author: Muhammad Anis Uddin Nasir

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>)

Tags: [2016](<https://devfeed.tech/tags/2016.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [fault-tolerance](<https://devfeed.tech/tags/fault-tolerance.md>), [flink](<https://devfeed.tech/tags/flink.md>), [high-availability](<https://devfeed.tech/tags/high-availability.md>), [latency](<https://devfeed.tech/tags/latency.md>), [logs](<https://devfeed.tech/tags/logs.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [query](<https://devfeed.tech/tags/query.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [replication](<https://devfeed.tech/tags/replication.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [spark](<https://devfeed.tech/tags/spark.md>), [storage](<https://devfeed.tech/tags/storage.md>), [storm](<https://devfeed.tech/tags/storm.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>), [study](<https://devfeed.tech/tags/study.md>), [survey](<https://devfeed.tech/tags/survey.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [upstream](<https://devfeed.tech/tags/upstream.md>)

### AI overview

The survey examines fault tolerance in distributed stream processing engines and how design choices relate to application requirements such as low latency, throughput, scalability, and availability. It covers S4, Storm, Flink, Spark Streaming, Samza, and MillWheel, organizing techniques as active replication, passive replication, and upstream backup. The supplied description says the revised version corrects errors in the 2016 manuscript, summarizes later Flink and Spark developments, and adds discussion of durable logs and archival storage for replay. The excerpt does not provide enough detail to compare individual engines' mechanisms or tradeoffs.

### Source excerpt

arXiv:1605.00928v4 Announce Type: replace Abstract: Distributed Stream Processing Engines (DSPEs) target applications related to continuous computation, online machine learning and real-time query processing. DSPEs operate on high volume of data by applying lightweight operations on real-time and continuous streams. Such systems require clusters of hundreds of machines for their deployment. Streaming applications come with various requirements, i.e., low-latency, high throughput, scalability and high availability. In this survey, we study the fault tolerance problem for DSPEs. We discuss fault tolerance techniques that are used in stream processing engines that were state of the art at the time of writing (2016), namely S4, Storm, Flink, Spark Streaming, Samza and MillWheel. Further, we give insight on fault tolerance approaches that we categorize as active replication, passive replication and upstream backup. Finally, we discuss implications of the fault tolerance techniques for different streaming application requirements. This revised version corrects errors in the original 2016 manuscript, summarizes later developments in Flink and Spark, discusses durable logs and archival storage for replay, and points to more recent surveys.

## Margin Certificates for Voting Classifiers under Mixing Dependence

DevFeed: [Margin Certificates for Voting Classifiers under Mixing Dependence](<https://devfeed.tech/articles/margin-certificates-for-voting-classifiers-under-mixing-dependence-93255.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.24166>)

Author: Adria Binte Habib, Ibne Farabi Shihab

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Occam's razor machine learning](<https://devfeed.tech/topics/occam-s-razor-machine-learning.md>), [Time Series](<https://devfeed.tech/topics/time-series.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>)

Tags: [certificates](<https://devfeed.tech/tags/certificates.md>), [classifiers](<https://devfeed.tech/tags/classifiers.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [margin](<https://devfeed.tech/tags/margin.md>), [math-st](<https://devfeed.tech/tags/math-st.md>), [mixing](<https://devfeed.tech/tags/mixing.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [stat-th](<https://devfeed.tech/tags/stat-th.md>), [voting](<https://devfeed.tech/tags/voting.md>)

### AI overview

The paper develops explicit voting-margin guarantees for dependent data. For stationary β-mixing streams and a finite base class, it combines independent-block Rademacher theory with clipped margin loss; the penalty depends on the block count and coupling budget 4(μ−1)β(a). An i.i.d. voting-margin guarantee also transfers to a regularly spaced subsequence with an additive β-mixing failure probability. For votes fixed independently of a reversible finite-state Markov stream, spectral concentration yields a √(t_rel/n) rate, and a sticky-refresh construction establishes that this exponent is unavoidable; the statistic-specific dependence scale is τ_int(g) ≤ 2t_rel−1. When dependence is unknown, a batch-means Student-t interval has an explicit remainder that vanishes with the number of batches. Reported experiments find that block-bootstrap and batch-means intervals undercover under strong dependence, while the certificate remains valid; on ordered UCI HAR sensor data, reported coverage is 0.95 for the certificate versus 0.38 for naive intervals. A two-timescale example reports a batch-means interval 8.1 times narrower than the worst-case interval at nominal coverage 0.945.

### Source excerpt

arXiv:2610.24166v1 Announce Type: new Abstract: Margin bounds explain why weighted votes generalize after training error vanishes, but dependent observations require concentration arguments that keep the dependence model and failure budget explicit. We study stationary mixing streams without treating dependence as a formal substitution of an "effective sample size." For a finite base class, we combine independent-block Rademacher theory for stationary $\beta$-mixing processes with a clipped margin loss to obtain an explicit uniform voting-margin certificate, whose dependence penalty is set by the number of approximately independent blocks and the coupling budget $4(\mu-1)\beta(a)$. A black-box transfer theorem shows that any i.i.d. voting-margin guarantee remains valid on a regularly spaced subsequence, with only an additive $\beta$-mixing failure probability. For votes fixed independently of a stationary finite-state reversible Markov stream, spectral concentration gives the cleaner $\sqrt{t_{rel}/n}$ rate, and a sticky-refresh construction shows this exponent is unavoidable. The constant is statistic-specific: the effective sample size is $n/\tau_{int}(g)$, with $\tau_{int}(g) \le 2t_{rel}-1$. When the dependence scale is unknown, a batch-means Student-t interval is valid up to an explicit remainder vanishing in the number of batches, with width governed by $\tau_{int}(g)$ rather than the worst-case $t_{rel}$. Experiments show: the learned-vote certificate is nonvacuous once the block count is large; a two-timescale construction gives a batch-means interval 8.1x narrower than the worst-case interval at nominal coverage (0.945); block-bootstrap and batch-means intervals undercover under strong dependence while the certificate stays valid; and on real, ordered UCI HAR sensor data the dependence-agnostic certificate reaches 0.95 coverage where naive intervals reach only 0.38.

## Monolithic 3D Memristor-TFT Stack For Programmable Neuromorphic Computing (SNU, Yonsei)

DevFeed: [Monolithic 3D Memristor-TFT Stack For Programmable Neuromorphic Computing (SNU, Yonsei)](<https://devfeed.tech/articles/monolithic-3d-memristor-tft-stack-for-programmable-neuromorphic-computing-snu-yonsei-66449.md>)

Original publisher: [Read original article](<https://semiengineering.com/monolithic-3d-memristor-tft-stack-for-programmable-neuromorphic-computing-snu-yonsei/>)

Author: Technical Paper Link

Published: 2026-10-06T21:29:31Z

Content type: article

Language: en

Sources: [Semiconductor Engineering](<https://devfeed.tech/sources/semiconductor-engineering.md>)

Topics: [Computing](<https://devfeed.tech/topics/computing.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Sustainable Computing](<https://devfeed.tech/topics/sustainable-computing.md>)

Tags: [3d-ic](<https://devfeed.tech/tags/3d-ic.md>), [ai-ml-dl](<https://devfeed.tech/tags/ai-ml-dl.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [back-end-of-line](<https://devfeed.tech/tags/back-end-of-line.md>), [beol](<https://devfeed.tech/tags/beol.md>), [computing](<https://devfeed.tech/tags/computing.md>), [emerging-memory](<https://devfeed.tech/tags/emerging-memory.md>), [hfo2](<https://devfeed.tech/tags/hfo2.md>), [in2o3](<https://devfeed.tech/tags/in2o3.md>), [memory](<https://devfeed.tech/tags/memory.md>), [memristors](<https://devfeed.tech/tags/memristors.md>), [monolithic-3d-integration](<https://devfeed.tech/tags/monolithic-3d-integration.md>), [neuromorphic](<https://devfeed.tech/tags/neuromorphic.md>), [neuromorphic-computing](<https://devfeed.tech/tags/neuromorphic-computing.md>), [packaging](<https://devfeed.tech/tags/packaging.md>), [reservoir-computing](<https://devfeed.tech/tags/reservoir-computing.md>), [resistive-memory](<https://devfeed.tech/tags/resistive-memory.md>), [seoul-national-university](<https://devfeed.tech/tags/seoul-national-university.md>), [technical-papers](<https://devfeed.tech/tags/technical-papers.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [temporal-computing](<https://devfeed.tech/tags/temporal-computing.md>), [tft](<https://devfeed.tech/tags/tft.md>), [thin-film-transistors](<https://devfeed.tech/tags/thin-film-transistors.md>), [transistors](<https://devfeed.tech/tags/transistors.md>), [yonsei-university](<https://devfeed.tech/tags/yonsei-university.md>)

### AI overview

Researchers at Seoul National University and Yonsei University report a monolithically integrated memristor-thin-film transistor reservoir kernel. The abstract says it enables electrically programmable temporal dynamics across a wide timescale, addressing limits in hardware reservoirs whose dynamics are fixed or narrowly tunable.

### Source excerpt

Researchers at Seoul National University and Yonsei University published a technical paper titled "Monolithic three-dimensionally integrated memristor-thin-film transistor for electrically programmable multimode reservoir computing." Abstract Excerpt: "Reservoir computing offers an efficient framework for processing temporal information, but most hardware reservoirs exhibit fixed or narrowly tunable dynamics determined by intrinsic material relaxation time. A monolithically integrated... " read more The post Monolithic 3D Memristor-TFT Stack For Programmable Neuromorphic Computing (SNU, Yonsei) appeared first on Semiconductor Engineering.

## Unlocking Real-Time Insights: Integrate KSQL with Kafka for Stream Processing

DevFeed: [Unlocking Real-Time Insights: Integrate KSQL with Kafka for Stream Processing](<https://devfeed.tech/articles/unlocking-real-time-insights-integrate-ksql-with-kafka-for-stream-processing-65418.md>)

Original publisher: [Read original article](<https://blog.devops.dev/unlocking-real-time-insights-integrate-ksql-with-kafka-for-stream-processing-d120ad646d20?source=rss----33f8b2d9a328---4>)

Author: Rahul Patidar

Published: 2026-10-06T13:04:14Z

Content type: tutorial

Language: en

Sources: [DevOps.dev](<https://devfeed.tech/sources/devops-dev.md>)

Topics: [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [declarative](<https://devfeed.tech/tags/declarative.md>), [fault-tolerance](<https://devfeed.tech/tags/fault-tolerance.md>), [filter](<https://devfeed.tech/tags/filter.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [integration](<https://devfeed.tech/tags/integration.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [join](<https://devfeed.tech/tags/join.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [ksql](<https://devfeed.tech/tags/ksql.md>), [ksqldb](<https://devfeed.tech/tags/ksqldb.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [processing](<https://devfeed.tech/tags/processing.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [sql](<https://devfeed.tech/tags/sql.md>), [stream](<https://devfeed.tech/tags/stream.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

The excerpt shows how Kafka message keys affect KSQL table results. It starts ZooKeeper, Kafka, and the KSQL server, then compares two transaction-topic examples: records produced without keys are displayed with null keys, while a console producer configured with `parse.key=true` and `key.separator=:` assigns keys. It creates KSQL streams and tables over each topic, plus queryable tables, and runs selects. The article also says to use `auto.offset.reset='earliest'` to read existing records or `'latest'` for new ones, and demonstrates `EMIT CHANGES` for continuing query output. It describes streams as continuously flowing records and tables as keyed state holding the latest values. The supplied text ends with the second scenario and a summary; it does not show a broader integration scenario.

### Source excerpt

What Is Kafka? Kafka is a powerful system that handles streaming data in real-time. It is like a highly reliable and scalable infrastructure that allows you to process and analyze large amounts of data as it flows through the system. With Kafka, you can build applications that handle data ingestion, analytics, and other real-time data processing tasks effectively. What Is KSQL? KSQL is a tool that allows you to use SQL-like queries to process and analyze streaming data in real-time using Apache Kafka. It simplifies the development of real-time applications by enabling you to write SQL queries to filter, transform, and join data streams, making it easier to work with and derive insights from streaming data. Why Do We Need Integration Between Kafka and Ksql? Real-time data processing: Enables the processing and analysis of streaming data in real-time. SQL-like interface: Allows working with streaming data using familiar SQL queries. Real-time analytics: Provides insights and analytics on continuously flowing data. Simplified development: Streamlines the development of real-time applications with a higher-level, declarative approach. Scalability and fault-tolerance: Supports handling high-volume streaming data with scalability and fault-tolerant processing. Key Components In Ksql : Stream: Represents continuously flowing real-time data for processing and analysis. Table: Organized collection of data records that result from aggregating or joining streams or tables. Maintains the latest values for each key. Queryable Table: A special table type allowing low-latency access to the current state for interactive querying. Pre-Requites : Start Zookeeper start kafka server start ksql servernohup ./bin/zookeeper-server-start ./etc/kafka/zookeeper.properties > zookeeper.log 2>&1 & nohup ./bin/kafka-server-start ./etc/kafka/server.properties > kafka.log 2>&1 & nohup ./bin/ksql-server-start ./etc/ksql/ksql-server.properties > ksql.log 2>&1 & How to Integrate Kafka with Ksql: Scen

## Real-Time Data for Real-Time AI: Designing Event-Driven Cloud Architectures for Autonomous Decision

DevFeed: [Real-Time Data for Real-Time AI: Designing Event-Driven Cloud Architectures for Autonomous Decision](<https://devfeed.tech/articles/real-time-data-for-real-time-ai-designing-event-driven-cloud-architectures-for-autonomous-decision-60589.md>)

Original publisher: [Read original article](<https://hackernoon.com/real-time-data-for-real-time-ai-designing-event-driven-cloud-architectures-for-autonomous-decision?source=rss>)

Author: Sunilkumar Reddy Eraganeni

Published: 2026-09-27T11:33:05Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [event-driven-ai-architecture](<https://devfeed.tech/tags/event-driven-ai-architecture.md>), [fraud-detection-architecture](<https://devfeed.tech/tags/fraud-detection-architecture.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kafka-stream-processing](<https://devfeed.tech/tags/kafka-stream-processing.md>), [low-latency-ai-systems](<https://devfeed.tech/tags/low-latency-ai-systems.md>), [point-in-time-correctness](<https://devfeed.tech/tags/point-in-time-correctness.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-time-ai-data-pipelines](<https://devfeed.tech/tags/real-time-ai-data-pipelines.md>), [real-time-ml-inference](<https://devfeed.tech/tags/real-time-ml-inference.md>), [streaming-ai-data-pipelines](<https://devfeed.tech/tags/streaming-ai-data-pipelines.md>)

### AI overview

The article explains how to design event-driven cloud systems for AI decisions with latency requirements. It stresses defining the decision's time budget, choosing streaming only when needed, and handling event-time processing issues such as late-arriving events and watermarks.

### Source excerpt

A model can be accurate and still make useless decisions if its data arrives too late. Learn how to build low-latency, event-driven AI systems.

## Partition Finalization in Pinterest's Next-Generation DB Ingestion Framework

DevFeed: [Partition Finalization in Pinterest's Next-Generation DB Ingestion Framework](<https://devfeed.tech/articles/partition-finalization-in-pinterest-s-next-generation-db-ingestion-framework-60046.md>)

Original publisher: [Read original article](<https://medium.com/pinterest-engineering/partition-finalization-in-pinterests-next-generation-db-ingestion-framework-4c7da6e4cc8f?source=rss----4c5a5f6279b6---4>)

Author: Pinterest Engineering

Published: 2026-09-25T15:01:03Z

Content type: article

Language: en

Sources: [Pinterest](<https://devfeed.tech/sources/pinterest-engineering-blog-medium.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [automated](<https://devfeed.tech/tags/automated.md>), [batch](<https://devfeed.tech/tags/batch.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [complete](<https://devfeed.tech/tags/complete.md>), [completeness](<https://devfeed.tech/tags/completeness.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [customers](<https://devfeed.tech/tags/customers.md>), [data](<https://devfeed.tech/tags/data.md>), [data-completeness](<https://devfeed.tech/tags/data-completeness.md>), [database](<https://devfeed.tech/tags/database.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [flink](<https://devfeed.tech/tags/flink.md>), [format](<https://devfeed.tech/tags/format.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generate](<https://devfeed.tech/tags/generate.md>), [generation](<https://devfeed.tech/tags/generation.md>), [hive](<https://devfeed.tech/tags/hive.md>), [icebergs](<https://devfeed.tech/tags/icebergs.md>), [next](<https://devfeed.tech/tags/next.md>), [partition](<https://devfeed.tech/tags/partition.md>), [pinterest](<https://devfeed.tech/tags/pinterest.md>), [spark](<https://devfeed.tech/tags/spark.md>)

### AI overview

Pinterest's streaming ingestion framework restores a reliable "safe to read" signal by deriving partition finalization from event time. The Flink-to-Iceberg sink records per-checkpoint event-time statistics in snapshot metadata, including compact percentile sketches, then computes a watermark from recent commits. Its default uses the minimum event time across a configurable lookback and rounds down to the hour; the watermark never regresses, and late data can trigger an alert. The CDC table stores the watermark as an Iceberg property, which Spark copies to the base table during upserts so downstream jobs can gate processing through metadata sensors. A conservative default favors completeness; tables can use a percentile policy for fresher data while accepting a rare late record. Generic accumulator and commit-processor extensions enable the mechanism in other Flink-to-Iceberg pipelines without extra operators or an external service.

### Source excerpt

Qianrui Zhang | Sr Software Engineer, Logging Platform Kanchi Masalia | Software Engineer II, Stream Processing Platform Liang Mou | Sr Staff Software Engineer, Logging Platform Yi Pan | Principal Engineer, Agent Platform Introduction This is the third post in our series on Pinterest's next-generation database ingestion framework. Part 1 introduced the DB ingestion framework built on Kafka, Flink, Spark, and Iceberg, and Part 2 covered automated schema evolution. This post tackles another challenge in migrating downstream customers to the new ingestion framework: knowing when data is complete enough to read. We'll walk through that data completeness challenge and how we solve it: what "partition finalization" means, why it matters to downstream customers, how we built a unified mechanism to generate partition finalization markers, and how those markers are consumed. We'll also look at how this mechanism generalizes beyond DB ingestion. Background & Motivation Before explaining partition finalization, it helps to define what a partition is and why a downstream consumer cares when one partition is "done." What is a Partition A time partition is a subset of a table whose rows are grouped by a time column, e.g. all rows whose timestamp falls within a given hour. Grouping data this way lets a consumer read just the slice it needs, such as the 1 AM hour, instead of scanning the whole table. How that grouping is physically represented depends on the table format: Hive uses explicit, directory-based partitioning: each partition maps to a physical directory in S3. Iceberg uses hidden partitioning: the partitioning is defined in the metadata layer rather than by directory layout, so consumers don't need to know the physical file layout to query a partition. What Finalization Means Now consider a typical consumer of these tables: a downstream batch job that processes a single hour's partition and runs only once for that hour. The consumer faces the question: when should my job

## Apache Flink Applications: Building Blocks for Stateful Stream Processing

DevFeed: [Apache Flink Applications: Building Blocks for Stateful Stream Processing](<https://devfeed.tech/articles/applications-48580.md>)

Original publisher: [Read original article](<https://flink.apache.org/what-is-flink/flink-applications/>)

Published: 2026-09-18T15:54:07.612713Z

Content type: tutorial

Language: en

Sources: [Apache Flink](<https://devfeed.tech/sources/apache-flink-stateful-computations-over-data-streams-on-apache-flink.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Stateful](<https://devfeed.tech/topics/stateful.md>), [Framework](<https://devfeed.tech/topics/framework.md>), [abstraction](<https://devfeed.tech/topics/abstraction.md>), [rocksdb](<https://devfeed.tech/topics/rocksdb.md>)

Tags: [abstraction](<https://devfeed.tech/tags/abstraction.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [framework](<https://devfeed.tech/tags/framework.md>), [processing](<https://devfeed.tech/tags/processing.md>), [rocksdb](<https://devfeed.tech/tags/rocksdb.md>), [stateful](<https://devfeed.tech/tags/stateful.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [streams](<https://devfeed.tech/tags/streams.md>)

### AI overview

This article introduces Apache Flink as a framework for stateful computations over bounded and unbounded data streams. It explains streaming application building blocks, including streams, state, time, APIs, libraries, and state backends such as RocksDB.

### Source excerpt

What is Apache Flink? -- Applications # Apache Flink is a framework for stateful computations over unbounded and bounded data streams. Flink provides multiple APIs at different levels of abstraction and offers dedicated libraries for common use cases. Here, we present Flink's easy-to-use and expressive APIs and libraries. Building Blocks for Streaming Applications # The types of applications that can be built with and executed by a stream processing framework are defined by how well the framework controls streams, state, and time.

## Code Style and Quality Guide -- Components Guide

DevFeed: [Code Style and Quality Guide -- Components Guide](<https://devfeed.tech/articles/code-style-and-quality-guide-components-guide-48563.md>)

Original publisher: [Read original article](<https://flink.apache.org/how-to-contribute/code-style-and-quality-components/>)

Published: 2026-09-18T15:54:07.612713Z

Content type: tutorial

Language: en

Sources: [Apache Flink](<https://devfeed.tech/sources/apache-flink-stateful-computations-over-data-streams-on-apache-flink.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [YAML](<https://devfeed.tech/topics/yaml.md>), [Java](<https://devfeed.tech/topics/java.md>), [Scala](<https://devfeed.tech/topics/scala.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [code](<https://devfeed.tech/tags/code.md>), [code-style](<https://devfeed.tech/tags/code-style.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [guide](<https://devfeed.tech/tags/guide.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [quality](<https://devfeed.tech/tags/quality.md>), [yaml](<https://devfeed.tech/tags/yaml.md>)

### AI overview

A guide to code style and quality practices for Apache Flink components. It covers pull requests, Java and Scala conventions, configuration placement and naming, connector design, concurrency and checkpointing considerations, and standards for useful examples.

### Source excerpt

Code Style and Quality Guide -- Components Guide # Preamble # Pull Requests & Changes # Common Coding Guide # Java Language Guide # Scala Language Guide # Components Guide # Formatting Guide # Component Specific Guidelines # Additional guidelines about changes in specific components. Configuration Changes # Where should the config option go? 'flink-conf.yaml': All configuration that pertains to execution behavior that one may want to standardize across jobs.

[Next page](<https://devfeed.tech/topics/stream-processing.md?cursor=WyIyMDI2LTA5LTE4VDE1OjU0OjA3LjYxMjcxMyswMDowMCIsICI2ZmU3Mzc0ZC1hZGEzLTQ5YTYtYjU3Ni00NWQyMDIzOTFiOTMiXQ%3D%3D>)