# streaming data analytics

Streaming data analytics is the continuous ingestion, processing, and analysis of streaming data in real time or near real time, enabling immediate insights from event-driven systems.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time

DevFeed: [ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time](<https://devfeed.tech/articles/arrivalbench-agent-generated-data-pipelines-are-correct-once-and-wrong-under-time-144954.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.02363>)

Author: Pranay Kothari

Published: 2026-11-20T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [data observability](<https://devfeed.tech/topics/data-observability.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [agent](<https://devfeed.tech/tags/agent.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [batch](<https://devfeed.tech/tags/batch.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [crash](<https://devfeed.tech/tags/crash.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [failure](<https://devfeed.tech/tags/failure.md>), [idempotency](<https://devfeed.tech/tags/idempotency.md>), [log](<https://devfeed.tech/tags/log.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [ordering](<https://devfeed.tech/tags/ordering.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

ArrivalBench tests agent-built data pipelines by replaying late, duplicated, out-of-order, and retried records, then comparing the final state with a batch recomputation of the full log. Unlike one-shot snapshot grading, this exposes silent wrong results separately from crashes. Across 40 tasks and eleven models, one-shot grading certified 86-100% of pipelines, while replay found 7.0-79.2% of those certified pipelines silently wrong. Idempotency hazards failed more often than ordering hazards in every model. In one model, a hazard warning reduced silent failures from 48.2% to 10.5%, but raised crashes from 9.0% to 37.0%; total failure fell only from 51.0% to 44.0%. Independently rerunning all eleven arms shifted rates by at most 5.9 percentage points.

### Source excerpt

arXiv:2610.02363v2 Announce Type: replace Abstract: Benchmarks for agent-generated data work grade a pipeline by running it once against a fixed snapshot. ArrivalBench instead re-executes the pipeline an agent leaves behind under adversarial but replayable delivery schedules (late, duplicated, out-of-order and retried records) and requires its final state to equal a batch recomputation of the complete log. Because the oracle recomputes rather than classifies, a wrong table and a crash are distinct verdicts: a crash is visible to monitoring a team already runs, and a wrong table is not. On 40 tasks we built, our reimplementation of single-execution grading certifies 86-100% of the pipelines eleven models produce; re-executing the same artifacts finds 7.0-79.2% of the certified ones silently wrong. The gap is not produced by the repair loop: within the same model and task, pipelines repaired against the snapshot test fail replay about as often as those that passed it first time. In every model, idempotency hazards fail more often than ordering hazards. Separating a wrong answer from a crash also changes how interventions read: a hazard warning cuts one model's silent failure from 48.2% to 10.5% while raising its crash rate from 9.0% to 37.0%, so all-in failure moves only from 51.0% to 44.0%. All eleven arms were independently re-run, and rates moved by at most 5.9 points.

## Continuous Prompts: LLM-Augmented Pipeline Processing over Unstructured Streams

DevFeed: [Continuous Prompts: LLM-Augmented Pipeline Processing over Unstructured Streams](<https://devfeed.tech/articles/continuous-prompts-llm-augmented-pipeline-processing-over-unstructured-streams-145730.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2512.03389>)

Author: Shu Chen, Deepti Raghavan, U\\u{g}ur \\c{C}etintemel

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [batching](<https://devfeed.tech/tags/batching.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-reasoning](<https://devfeed.tech/tags/llm-reasoning.md>), [long-running](<https://devfeed.tech/tags/long-running.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [operator](<https://devfeed.tech/tags/operator.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [probing](<https://devfeed.tech/tags/probing.md>), [queries](<https://devfeed.tech/tags/queries.md>), [rag](<https://devfeed.tech/tags/rag.md>), [stream](<https://devfeed.tech/tags/stream.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>), [streams](<https://devfeed.tech/tags/streams.md>)

### AI overview

Continuous Prompts (CPs) brings persistent LLM reasoning to stream processing by extending retrieval-augmented generation to streaming data and defining continuous semantic operators. Implemented in VectraFlow, it offers multiple implementations, mainly LLM-based plus an embedding-based variant. Tuple batching and operator fusion improve efficiency at the cost of accuracy. To manage that trade-off, CPs uses lightweight shadow executions and cost-aware multi-objective Bayesian optimization to learn throughput-accuracy frontiers and adapt plans within probing budgets. The abstract reports evaluations with operator microbenchmarks and real-dataset pipelines, finding that VectraFlow adapts to workload changes and sustains semantic queries over evolving unstructured streams.

### Source excerpt

arXiv:2512.03389v2 Announce Type: replace Abstract: Monitoring unstructured streams increasingly requires persistent, semantics-aware computation, yet today's LLM frameworks remain stateless and one-shot, limiting their usefulness for long-running analytics. We introduce Continuous Prompts (CPs), the first framework that brings LLM reasoning into continuous stream processing. CPs extend RAG to streaming settings, define continuous semantic operators, and provide multiple implementations, primarily focusing on LLM-based approaches but also reporting one embedding-based variants. Furthermore, we study two LLM-centric optimizations, tuple batching and operator fusion, to significantly improve efficiency while managing accuracy loss. Because these optimizations inherently trade accuracy for speed, we present a dynamic optimization framework that uses lightweight shadow executions and cost-aware multi-objective Bayesian optimization (MOBO) to learn throughput-accuracy frontiers and adapt plans under probing budgets. We implement CPs in the VectraFlow stream processing system. Using operator-level microbenchmarks and streaming pipelines on real datasets, we show that VectraFlow can adapt to workload dynamics, navigate accuracy-efficiency trade-offs, and sustain persistent semantic queries over evolving unstructured streams.

## Enabling Real-Time AI for Scientific Instruments with GPU-Native Data Acquisition

DevFeed: [Enabling Real-Time AI for Scientific Instruments with GPU-Native Data Acquisition](<https://devfeed.tech/articles/enabling-real-time-ai-for-scientific-instruments-with-gpu-native-data-acquisition-143888.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/enabling-real-time-ai-for-scientific-instruments-with-gpu-native-data-acquisition/>)

Author: Cara Laasch

Published: 2026-11-19T17:00:00Z

Content type: article

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [Holoscan](<https://devfeed.tech/topics/holoscan.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [acquisition](<https://devfeed.tech/tags/acquisition.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [batch](<https://devfeed.tech/tags/batch.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [communication](<https://devfeed.tech/tags/communication.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [connectx](<https://devfeed.tech/tags/connectx.md>), [control](<https://devfeed.tech/tags/control.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [data](<https://devfeed.tech/tags/data.md>), [developer-tools-techniques](<https://devfeed.tech/tags/developer-tools-techniques.md>), [dgx-spark](<https://devfeed.tech/tags/dgx-spark.md>), [edge-computing](<https://devfeed.tech/tags/edge-computing.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [instruments](<https://devfeed.tech/tags/instruments.md>), [measurements](<https://devfeed.tech/tags/measurements.md>), [networking-communications](<https://devfeed.tech/tags/networking-communications.md>), [processes](<https://devfeed.tech/tags/processes.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [sc26](<https://devfeed.tech/tags/sc26.md>), [science](<https://devfeed.tech/tags/science.md>), [sensor](<https://devfeed.tech/tags/sensor.md>)

### AI overview

NVIDIA DAQIRI moves high-bandwidth sensor data directly into GPU memory so it can be processed during acquisition, enabling real-time inspection and feedback without first writing measurements to disk. At Lawrence Berkeley National Laboratory's Molecular Foundry, a high-dynamic-range electron microscopy camera streams over 500 Gbps; DAQIRI and GPU processing assemble and correct detector frames, derive energy-loss spectra, and improve spatial resolution from 5-micrometer sensor pixels to submicron resolution. The latest release adds packet reordering on supported NVIDIA ConnectX-7-or-newer NICs, an ibverbs backend, and direct polling. The ibverbs engine benchmarks at about twice the previous DPDK backend's per-core throughput for packets up to 1,024 bytes at rates above 200 Gbps on specified NVIDIA platforms. Direct polling avoids a separate I/O-worker handoff; cabled loopback tests demonstrated roughly 2-4 μs application send-to-receive latency. These results are benchmark-specific, and the excerpt recommends evaluating throughput and latency on the target hardware.

### Source excerpt

In science and industry, analyzing sensor measurements in real time can enable AI to adjust processes, guide experiments, and search for better operating...

## Doing More by Storing Less: Speeding Up Real-Time Machine Learning

DevFeed: [Doing More by Storing Less: Speeding Up Real-Time Machine Learning](<https://devfeed.tech/articles/doing-more-by-storing-less-speeding-up-real-time-machine-learning-144308.md>)

Original publisher: [Read original article](<https://medium.com/feedzaitech/doing-more-by-storing-less-speeding-up-real-time-machine-learning-23639ae00e22?source=rss----e11168e7fe6b---4>)

Author: Augusto Peres

Published: 2026-11-19T15:40:42Z

Content type: article

Language: en

Sources: [Feedzai](<https://devfeed.tech/sources/feedzai.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>)

Tags: [aggregate](<https://devfeed.tech/tags/aggregate.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [counts](<https://devfeed.tech/tags/counts.md>), [database](<https://devfeed.tech/tags/database.md>), [detection](<https://devfeed.tech/tags/detection.md>), [event](<https://devfeed.tech/tags/event.md>), [financial-applications](<https://devfeed.tech/tags/financial-applications.md>), [fraud-detection](<https://devfeed.tech/tags/fraud-detection.md>), [heavy-hitter](<https://devfeed.tech/tags/heavy-hitter.md>), [inference](<https://devfeed.tech/tags/inference.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [latency](<https://devfeed.tech/tags/latency.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

Feedzai's event filtering reduces costly read-modify-write database cycles for high-volume entities while aiming to preserve useful feature profiles. Below a chosen per-entity write-rate threshold, every event is stored; above it, events are randomly included with probability set by the threshold divided by the estimated arrival rate. Kept events are weighted by the inverse of that probability to compensate counts and sums for discarded updates. The system can estimate arrival rates from filtered persisted statistics, updating and scaling those estimates through the same sampling process rather than tracking every entity's rate in memory; the article describes this as self-correcting. It says the approach fits power-law transaction volumes and reports that model accuracy did not decline under heavy filtering, with small gains in some cases attributed to training noise. The excerpt provides no quantitative performance results.

### Source excerpt

Modern fraud detection systems rely heavily on tracking behavior over time. For example: How much money has a person spent this month? Or, how many purchases have they made in the last week? Any event-streaming system worth its salt tracks hundreds of such metrics. Every time a new transaction even arrives, these are retrieved from a database, updated, and then written back to persistent storage. This process is known as the read-modify-write cycle, and its final write stage is a major source of latency, database contention, and infrastructure cost. But here is the catch: In many financial applications, a small set of heavy-hitter entities triggers most of these cycles and accounts for a disproportionate number of transactions. Statistically, the more events we process for any entity, the less information each individual event adds to their overall profile. Thus, here at Feedzai we developed a process and system that automatically stops writing every single event for large financial entities, saving significant compute without losing downstream ML accuracy. A friendly introduction to feature profiles Before we go any deeper into discarding events, we briefly explain what feature profiles are. Essentially, feature profiles aggregate events over a period of time. Two common examples are: Counts: How many times have we seen an entity doing something? e.g., the number of transactions for a specific debit card in the last month. Sums: The running total of a quantity associated with an event, e.g., the total spend of a user in the last week. In a real-time fraud detection system, when a single event arrives, the feature profiles for its associated entity must be retrieved from persistent storage, (e.g., a relational database or key-value store). Then, the profiles must be updated with the information on the new event. Next, the updated profiles are sent to a Machine Learning inference service for classification (e.g., fraud or not fraud) and written back to persistent sto

## StreamPhy: Streaming Inference of High-Dimensional Physical Dynamics via State Space Models

DevFeed: [StreamPhy: Streaming Inference of High-Dimensional Physical Dynamics via State Space Models](<https://devfeed.tech/articles/streamphy-streaming-inference-of-high-dimensional-physical-dynamics-via-state-space-models-143254.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.07384>)

Author: Panqi Chen, Yifan Sun, Shikai Fang, Xiao Fu, Lei Cheng

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Mamba](<https://devfeed.tech/topics/mamba.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [encoder](<https://devfeed.tech/tags/encoder.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [faster](<https://devfeed.tech/tags/faster.md>), [field](<https://devfeed.tech/tags/field.md>), [fields](<https://devfeed.tech/tags/fields.md>), [framework](<https://devfeed.tech/tags/framework.md>), [functional](<https://devfeed.tech/tags/functional.md>), [generation](<https://devfeed.tech/tags/generation.md>), [generative](<https://devfeed.tech/tags/generative.md>), [generative-models](<https://devfeed.tech/tags/generative-models.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [inference](<https://devfeed.tech/tags/inference.md>), [models](<https://devfeed.tech/tags/models.md>), [state-space](<https://devfeed.tech/tags/state-space.md>), [state-space-models](<https://devfeed.tech/tags/state-space-models.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

StreamPhy estimates evolving physical fields in real time from irregular sparse measurements. It combines an observation-pattern-adaptive encoder, a structured state-space model for memory-efficient updates across irregular time intervals, and an FT-FiLM decoder that generates continuous fields. The authors prove FT-FiLM can represent a richer function class than the functional Tucker model. In experiments on three physical systems with challenging sampling patterns, they report at least 48% better accuracy than state-of-the-art baselines and inference 20-100 times faster than diffusion-based methods. The available text is an abstract and does not specify the systems, evaluation setup, or comparisons behind these results.

### Source excerpt

arXiv:2605.07384v4 Announce Type: replace Abstract: Inferring the evolution of high-dimensional and multi-modal (e.g., spatio-temporal) physical fields from irregular sparse measurements in real time is a fundamental challenge in science and engineering. Existing approaches, including diffusion-based generative models and functional tensor methods, typically operate in offline settings, depend on full temporal observations, or incur substantial inference cost. We propose StreamPhy, an end-to-end framework that enables efficient and accurate streaming inference of full-field physical dynamics from incoming irregular sparse measurements. The framework integrates a data-adaptive observation encoder that is robust to arbitrary observation patterns, a structured state-space model that supports memory-efficient online updates across irregular time intervals, and an expressive Functional Tensor Feature-wise Linear Modulation (FT-FiLM) decoder for continuous-field generation. We prove that FT-FiLM is more expressive than the functional Tucker model, admitting a richer function class for handling complex dynamics. Experiments on three representative physical systems under challenging sampling patterns show that StreamPhy consistently outperforms state-of-the-art baselines, with at least 48\% improvement in accuracy and up to 20--100X faster inference than diffusion-based methods.

## Aderant modernizes analytics data pipeline with Apache Iceberg and AWS Lambda

DevFeed: [Aderant modernizes analytics data pipeline with Apache Iceberg and AWS Lambda](<https://devfeed.tech/articles/aderant-modernizes-analytics-data-pipeline-with-apache-iceberg-and-aws-lambda-141916.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/aderant-modernizes-analytics-data-pipeline-with-apache-iceberg-and-aws-lambda/>)

Author: Sham Birbahadur

Published: 2026-11-19T00:36:23Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [event-driven data processing](<https://devfeed.tech/topics/event-driven-data-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [AWS Database Migration Service](<https://devfeed.tech/topics/aws-database-migration-service.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [acid](<https://devfeed.tech/tags/acid.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [batch](<https://devfeed.tech/tags/batch.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cost](<https://devfeed.tech/tags/cost.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [faster](<https://devfeed.tech/tags/faster.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [legal](<https://devfeed.tech/tags/legal.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [queries](<https://devfeed.tech/tags/queries.md>)

### AI overview

Aderant replaced Stridyn's 15-minute batch pipeline with an event-driven medallion architecture that processes CDC changes from 700 client environments in single-digit seconds to bronze-layer Apache Iceberg tables. AWS DMS delivers Parquet files to S3; S3 notifications and SQS invoke Lambda for ingestion, while scheduled AWS Glue jobs transform data through silver and gold layers. Incremental Glue jobs handle 77 standard gold tables; 34 complex tables use Iceberg materialized views and snapshot comparisons to send changed rows to native Amazon Redshift tables. Aderant reports 10-26x faster queries on native Redshift tables, under-100-millisecond dashboard queries, and 77 percent lower pipeline compute costs versus streaming alternatives. The account says this approach supports schema changes and concurrent tenant workloads; it also describes a possible future Kinesis path for high-volume tenants.

### Source excerpt

Learn how Aderant modernized the data pipeline behind its Stridyn legal analytics platform with Apache Iceberg, AWS Lambda, and AWS Glue, moving from 15-minute batch cycles to an event-driven medallion architecture that processes change data capture events in single-digit seconds for 700 tenants, with 10-26x faster queries and 77 percent lower cost.

## How United Airlines streams database changes to Apache Iceberg with Amazon Data Firehose

DevFeed: [How United Airlines streams database changes to Apache Iceberg with Amazon Data Firehose](<https://devfeed.tech/articles/how-united-airlines-streams-database-changes-to-apache-iceberg-with-amazon-data-firehose-141918.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/how-united-airlines-streams-database-changes-to-apache-iceberg-with-amazon-data-firehose/>)

Author: Guillaume Dupont

Published: 2026-11-18T20:37:39Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Amazon Aurora](<https://devfeed.tech/topics/amazon-aurora.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Visual Studio Code](<https://devfeed.tech/topics/visual-studio-code.md>)

Tags: [airlines](<https://devfeed.tech/tags/airlines.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [amazon-data-firehose](<https://devfeed.tech/tags/amazon-data-firehose.md>), [amazon-kinesis](<https://devfeed.tech/tags/amazon-kinesis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cloudformation](<https://devfeed.tech/tags/aws-cloudformation.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [dms](<https://devfeed.tech/tags/dms.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [kinesis-data-streams](<https://devfeed.tech/tags/kinesis-data-streams.md>), [streams](<https://devfeed.tech/tags/streams.md>), [united-airlines](<https://devfeed.tech/tags/united-airlines.md>)

### AI overview

United Airlines describes a near-real-time CDC pipeline that moves Aurora MySQL changes through AWS DMS, Kinesis Data Streams, Lambda, and Amazon Data Firehose into Apache Iceberg tables on S3. Firehose applies inserts, updates, and deletes as Iceberg snapshots; the open tables can be queried through AWS Glue, Redshift, or Athena. United reports 5x lower average query latency and 2x shorter downstream Glue job processing time in its testing, with a low-volume test reaching about 10 seconds end-to-end. Buffer settings trade latency against cost and file size. The walkthrough uses DMS Serverless, Athena for validation, and a simplified single-table CloudFormation template; United's production setup uses provisioned DMS and primarily Redshift Serverless. The template's shard count and other demo settings are not production-tuned, and schema evolution and multi-table replication require additional handling.

### Source excerpt

United Airlines built a near-real-time change data capture pipeline that replicates reference data from Amazon Aurora MySQL to open Apache Iceberg tables using AWS DMS, Amazon Kinesis Data Streams, and Amazon Data Firehose. This post shows how to reproduce the serverless, multi-engine analytics pattern with AWS CloudFormation.

## Time-Series Models in the Streamhouse: A New Frontier and Use Cases

DevFeed: [Time-Series Models in the Streamhouse: A New Frontier and Use Cases](<https://devfeed.tech/articles/time-series-models-in-the-streamhouse-a-new-frontier-and-use-cases-143009.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/ibm-research/time-series-powered-streamhouse>)

Author: Nicholas Fuller; Sean Falconer; Ayhan Sebin; Pankaj Dayama; Roman Vaculin; Bilge Zeren Aksu; Julian Payne; Mountu Jinwala; Wesley M Gifford; Chandra Reddy; Vijay Ekambaram; Arindam Jati; Robert Metzge

Published: 2026-11-18T15:00:23Z

Content type: release

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Time Series](<https://devfeed.tech/topics/time-series.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Confluent Platform](<https://devfeed.tech/topics/confluent-platform.md>)

Tags: [anomaly-detection](<https://devfeed.tech/tags/anomaly-detection.md>), [business](<https://devfeed.tech/tags/business.md>), [can](<https://devfeed.tech/tags/can.md>), [cases](<https://devfeed.tech/tags/cases.md>), [change](<https://devfeed.tech/tags/change.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [confluent](<https://devfeed.tech/tags/confluent.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cpus](<https://devfeed.tech/tags/cpus.md>), [detection](<https://devfeed.tech/tags/detection.md>), [downloads](<https://devfeed.tech/tags/downloads.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [form](<https://devfeed.tech/tags/form.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [help](<https://devfeed.tech/tags/help.md>), [models](<https://devfeed.tech/tags/models.md>), [series](<https://devfeed.tech/tags/series.md>), [time](<https://devfeed.tech/tags/time.md>), [time-series](<https://devfeed.tech/tags/time-series.md>), [use-cases](<https://devfeed.tech/tags/use-cases.md>)

### AI overview

IBM time-series foundation models are available as native Confluent functions in managed Confluent Cloud and on-premises Confluent Platform for Apache Flink, running on CPUs. Flink can apply forecasting, anomaly detection, and time-series representations to streaming data with SQL, putting results alongside current events. Pretrained models offer a starting point across many series; workflows can incorporate recent observations, business rules, and human review. Examples include forecasting production quality to evaluate setpoints, flagging unusual store sales against a chosen confidence level, refreshing store-level demand forecasts to inform reorder points, adding individual spending patterns to existing payment checks, and pacing trades from short-term volume forecasts. Similarity search with TSPulse requires a separately built embedding pipeline and index, and can help investigate patterns but cannot establish root cause. These scenarios are illustrative; outcomes depend on model requirements, pipeline configuration, and application logic.

### Source excerpt

**Real-time forecasting, anomaly detection, and optimization, powered by IBM time-series models with more than 50 million downloads. Now available natively on Confluent, in the cloud and on premises, running on CPUs.** Every business generates a continuous record of change: a machine reading, a payment, a sale, an order, or a change in inventory. These observations form time series, and they carry signals that can help teams anticipate what happens next.

## Trends that will dominate analytics & BI in 2022 | Blog | Fivetran

DevFeed: [Trends that will dominate analytics & BI in 2022 | Blog | Fivetran](<https://devfeed.tech/articles/trends-that-will-dominate-analytics-bi-in-2022-blog-fivetran-141345.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/trends-that-will-dominate-analytics-bi-2022>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [DataOps](<https://devfeed.tech/topics/dataops.md>), [elt](<https://devfeed.tech/topics/elt.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [etl](<https://devfeed.tech/topics/etl.md>), [databricks](<https://devfeed.tech/topics/databricks.md>)

Tags: [2022](<https://devfeed.tech/tags/2022.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [automation](<https://devfeed.tech/tags/automation.md>), [beam](<https://devfeed.tech/tags/beam.md>), [bi](<https://devfeed.tech/tags/bi.md>), [bi-tools](<https://devfeed.tech/tags/bi-tools.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [boost](<https://devfeed.tech/tags/boost.md>), [building](<https://devfeed.tech/tags/building.md>), [business](<https://devfeed.tech/tags/business.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [chaos](<https://devfeed.tech/tags/chaos.md>), [dataops](<https://devfeed.tech/tags/dataops.md>), [elt](<https://devfeed.tech/tags/elt.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [managed](<https://devfeed.tech/tags/managed.md>), [trends](<https://devfeed.tech/tags/trends.md>)

### AI overview

The article predicts five shifts in analytics and BI for 2022: ELT pipelines becoming more continuous and fully managed, reducing reliance on brittle batch ETL; DataOps practices and features becoming more central, with dbt Labs cited for SQL-based warehouse transformations, lineage tracking, and version control; operational analytics making data available in frontline tools to support day-to-day decisions; BI tools enabling more collaborative work between data and business teams; and wider use of distributed processing frameworks such as Apache Spark, Beam, Flink, and PySpark. These are presented as forecasts, not measured outcomes.

### Source excerpt

Fully managed ELT, DataOps and more trends that will change the way we use data this year.

## CheckCo: Checkpoint-Coordinated Load Balancingfor Skewed Event Streams at Scale

DevFeed: [CheckCo: Checkpoint-Coordinated Load Balancingfor Skewed Event Streams at Scale](<https://devfeed.tech/articles/checkco-checkpoint-coordinated-load-balancingfor-skewed-event-streams-at-scale-140388.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22476>)

Author: Muhammad Anis Uddin Nasir, Reza Sadraei

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [checkpoint](<https://devfeed.tech/tags/checkpoint.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [event-streams](<https://devfeed.tech/tags/event-streams.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>), [routing](<https://devfeed.tech/tags/routing.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [workers](<https://devfeed.tech/tags/workers.md>)

### AI overview

CheckCo uses checkpoint barriers to adapt routing to skewed event streams without prior key-frequency statistics. Workers count events per key locally; at each barrier, counts are merged to split keys roughly in proportion to their estimated frequency and pack those splits onto workers for the next checkpoint window. The authors derive bounds on load imbalance and total key fan-out. On a production pipeline handling over 100 billion events daily, the most-loaded worker received 9.8% more events than average over a day (imbalance ratio 1.10), compared with 8.03 for key-based routing in simulation. In simulation, CheckCo used less than half the fan-out of earlier partitioners with comparable balance. Routing reflects distribution shifts from the next checkpoint, and the authors report balanced load through job restarts.

### Source excerpt

arXiv:2611.22476v1 Announce Type: new Abstract: Event streams are rarely uniform: a few dominant keys can generate orders of magnitude more events than the rest, overloading the workers that receive them and capping throughput. We present CheckCo, a checkpoint-coordinated load-balancing algorithm built on a dynamic oracle that estimates per-key frequencies from the live stream, with no prior statistics. Workers count events per key locally; at each checkpoint barrier, the counts are merged into a partitioning function that divides each key into a number of splits roughly proportional to its estimated frequency and packs the splits onto workers, and this function routes the next checkpoint window. We derive approximation bounds on the resulting load imbalance from classical scheduling results and bound its total key fan-out. On a production pipeline processing over 100 billion events per day, the most loaded worker receives only 9.8% more events than the average over a day (imbalance ratio 1.10), against 8.03 for key-based routing in simulation; in simulation, CheckCo also needs less than half the fan-out of earlier partitioners that reach comparable balance. Shifts in the key distribution are reflected in routing from the next checkpoint on, and the load stays balanced through job restarts.

## A Temporal Algebraic Framework for Data Stream Evaluation Under Delayed Labels and Distribution Shifts

DevFeed: [A Temporal Algebraic Framework for Data Stream Evaluation Under Delayed Labels and Distribution Shifts](<https://devfeed.tech/articles/a-temporal-algebraic-framework-for-data-stream-evaluation-under-delayed-labels-and-distribution-shifts-139926.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21914>)

Author: Ronaldo C. Prati, Elaine Ribeiro de Faria

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Lean](<https://devfeed.tech/topics/lean.md>)

Tags: [computer-science](<https://devfeed.tech/tags/computer-science.md>), [concept-drift](<https://devfeed.tech/tags/concept-drift.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [lean-4](<https://devfeed.tech/tags/lean-4.md>), [mask](<https://devfeed.tech/tags/mask.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [package](<https://devfeed.tech/tags/package.md>), [precision](<https://devfeed.tech/tags/precision.md>), [recall](<https://devfeed.tech/tags/recall.md>), [sliding-window](<https://devfeed.tech/tags/sliding-window.md>), [specificity](<https://devfeed.tech/tags/specificity.md>), [stream](<https://devfeed.tech/tags/stream.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [temporal](<https://devfeed.tech/tags/temporal.md>)

### AI overview

The framework extends confusion-matrix-based micro and macro evaluation to data streams using exponentially faded or sliding-window counts, with O(m²) updates per step. It handles delayed or selectively observed labels by temporarily counting instances through calibrated class probabilities; interim bias depends on calibration error, and classes without verified instances receive no value under audited conventions. To distinguish prevalence shifts from concept drift, it monitors recall and specificity and uses a first-order Jacobian decomposition to attribute metric changes. On synthetic streams and four real datasets, a detector using these prior-free coordinates and per-class memory produced no excess false alarms under label shift, unlike the Jacobian alarm and error-rate detectors. Delayed-label correction recovered recall in some real settings but only partially corrected precision, specificity, and F1. The abstract reports 25 statements checked in Lean 4 and code in the aggmetrics package.

### Source excerpt

arXiv:2611.21914v1 Announce Type: cross Abstract: In data streams, labels often arrive late or only for some instances, class prevalence drifts, and rare classes vanish from short windows. Standard evaluation then confuses a degraded decision boundary with a change in prevalence, and is biased when labels are delayed or selected by the model. We extend an algebraic framework that obtains pooled (micro) and per-class (macro) measures from one confusion matrix to streams, weighting the counts in time by exponential fading or a sliding window; the update costs O(m^2) per step, whatever the stream length, and each pooled measure stays a weighted combination of the per-class ones. Delayed and selective feedback enter as an observation mask: until its label arrives, an instance is counted with the model's calibrated class probabilities, and the bias of this interim estimate is driven by their calibration error. Audited conventions report no value for a class without verified instances. We separate prior shift from concept drift by monitoring prior-free coordinates (recall and specificity) and attribute a change in a measure to the two sources with a first-order Jacobian decomposition. On synthetic streams and four real datasets, a detector on these coordinates with a memory per class raises no excess false alarms under label shift, unlike the Jacobian alarm and error-rate detectors. The correction for delayed labels recovers recall in some real settings and is only partial for precision, specificity and F1. Twenty-five statements are checked in Lean 4, and the code is in the aggmetrics package.

## How to enable smart manufacturing with AI-powered insights | Blog | Fivetran

DevFeed: [How to enable smart manufacturing with AI-powered insights | Blog | Fivetran](<https://devfeed.tech/articles/how-to-enable-smart-manufacturing-with-ai-powered-insights-blog-fivetran-140238.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/how-to-enable-smart-manufacturing-with-ai-powered-insights>)

Author: Fivetran

Published: 2026-11-18T04:00:45Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [decision-making](<https://devfeed.tech/topics/decision-making.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [Internet of things](<https://devfeed.tech/topics/iot.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [manufacturing](<https://devfeed.tech/tags/manufacturing.md>), [smart-manufacturing](<https://devfeed.tech/tags/smart-manufacturing.md>)

### AI overview

Manufacturers can use AI for forecasting and generative tasks, but fragmented data and manual processes hinder adoption. The article describes supply chain command centers as hubs that combine equipment, analytics, automation and business processes to support end-to-end monitoring, disruption alerts, scenario modeling and routine decision automation. Tredence provides data architecture, modernization and AI capabilities; Fivetran supplies automated ingestion from more than 700 sources into analytics platforms. In an aluminum plant example, the partners streamed historian data and ingested batch quality data from edge to cloud, developed predictive models with Databricks and Microsoft Azure, then deployed them at the plant edge. The company reported 23% fewer production stoppages, 17% less work-in-process scrap and about 7% improved product quality.

### Source excerpt

Connect data from across all sources to see, understand, and predict every aspect of manufacturing operations and optimize decision making.

## Reduce the complexity and cost of real-time processing with queues in Amazon Kinesis Data Streams

DevFeed: [Reduce the complexity and cost of real-time processing with queues in Amazon Kinesis Data Streams](<https://devfeed.tech/articles/reduce-the-complexity-and-cost-of-real-time-processing-with-queues-in-amazon-kinesis-data-streams-140167.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/reduce-the-complexity-and-cost-of-real-time-processing-with-queues-in-amazon-kinesis-data-streams/>)

Author: Dirk Fröhner

Published: 2026-11-17T23:17:16Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-kinesis](<https://devfeed.tech/tags/amazon-kinesis.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [backlog](<https://devfeed.tech/tags/backlog.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [demand](<https://devfeed.tech/tags/demand.md>), [instead](<https://devfeed.tech/tags/instead.md>), [kinesis](<https://devfeed.tech/tags/kinesis.md>), [kinesis-data-streams](<https://devfeed.tech/tags/kinesis-data-streams.md>), [offers](<https://devfeed.tech/tags/offers.md>), [one](<https://devfeed.tech/tags/one.md>), [processing](<https://devfeed.tech/tags/processing.md>), [queues](<https://devfeed.tech/tags/queues.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [streams](<https://devfeed.tech/tags/streams.md>), [time](<https://devfeed.tech/tags/time.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Amazon Kinesis Data Streams queues provide per-record delivery tracking for applications that can process records independently. Consumers receive batches of up to 10 records, process them, and acknowledge them; unacknowledged records become available again after a configurable visibility timeout. Delivery is at least once and ordering is best effort, so processing should be idempotent. Records that exceed the default six receives without acknowledgment go to a dead-letter state, where operators can inspect, restore, or discard them. Queues scale consumer instances with backlog rather than shard count and remove the need for a KCL lease table and per-shard custom metrics. Multiple queues, stream consumers, and managed delivery can use the same stream without a relay or duplicate ingest. Use direct stream consumers when processing depends on record order or prior state; use queues for independent work. AWS reports up to 43% lower processing cost, but its synthetic benchmark used an I/O-bound handler; savings depend on workload and configuration, and CPU-bound work narrows the compute benefit.

### Source excerpt

Amazon Kinesis Data Streams now offers queues, a new way to consume the records of an on-demand stream. A queue delivers records one by one, tracks each record's delivery state, redelivers unacknowledged records, sets persistently failing records aside as dead letters, and scales with your backlog instead of your shard count.

## Smooth Cruiser at Lyft: Built for Safety and Transparency

DevFeed: [Smooth Cruiser at Lyft: Built for Safety and Transparency](<https://devfeed.tech/articles/smooth-cruiser-at-lyft-built-for-safety-and-transparency-140348.md>)

Original publisher: [Read original article](<https://eng.lyft.com/smooth-cruiser-at-lyft-built-for-safety-and-transparency-b6a461e16f2a?source=rss----25cd379abb8---4>)

Author: Lyft Eng Blog Admin

Published: 2026-11-17T14:27:11Z

Content type: article

Language: en

Sources: [Lyft Engineering](<https://devfeed.tech/sources/lyft-engineering-medium.md>)

Topics: [responsible-ai](<https://devfeed.tech/topics/responsible-ai.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [User interface design](<https://devfeed.tech/topics/ui-design.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [availability](<https://devfeed.tech/tags/availability.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [context](<https://devfeed.tech/tags/context.md>), [data](<https://devfeed.tech/tags/data.md>), [drivers](<https://devfeed.tech/tags/drivers.md>), [driving](<https://devfeed.tech/tags/driving.md>), [lyft](<https://devfeed.tech/tags/lyft.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [markets](<https://devfeed.tech/tags/markets.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [risk](<https://devfeed.tech/tags/risk.md>), [safety](<https://devfeed.tech/tags/safety.md>), [score](<https://devfeed.tech/tags/score.md>), [signal](<https://devfeed.tech/tags/signal.md>), [telematics](<https://devfeed.tech/tags/telematics.md>), [transparency](<https://devfeed.tech/tags/transparency.md>)

### AI overview

Lyft's Smooth Cruiser score evaluates driving relative to local road and time conditions rather than fixed universal thresholds. It tracks smooth braking, smooth steering, vehicle speed, and mounted-phone use only while a passenger is in the car; scores compare Drivers with peers in regions grouped by event-rate characteristics. Drivers qualify after 20 rides within 12 months, and scores use their latest 100 rides from the past 12 months to balance stability with responsiveness. Lyft's pipeline detects events from phone telemetry, enriches rides with location-based expected risk, and computes scores daily; results are stored in DynamoDB for app retrieval. Score Details Pages show weekly trends and the rides contributing to each behavior score, with rides appearing the following day.

### Source excerpt

At Lyft, we want to ensure everyone on the Lyft platform is supported by smart, fair tools to improve their experience. This is the guiding principle behind Smooth Cruiser, our feature that provides Drivers with insights into their driving behavior. For a tool like this to truly help Drivers make informed decisions for themselves, it must excel in two areas: real-world scoring and increased transparency into how that score is determined. Safety scoring often relies on fixed thresholds -- a single cutoff applied to every Driver, on every road, in every market. This method fails to account for the dynamic reality of driving. A harsh brake on a quiet local road is fundamentally different from a sudden stop in city traffic. Drivers need a system that recognizes these nuances. Smooth Cruiser recognizes these nuances and is built on two principles: a) score that reflects real world conditions, and b) transparency. In Part 1, we break down how the scoring works -- how every signal is measured relative to local conditions, how Drivers are benchmarked against peers in structurally similar markets, and how a rolling window captures typical behavior rather than isolated moments. In Part 2, we walk through how we brought that methodology to the surface -- giving Drivers direct visibility into the data behind their score, ride by ride, so they can understand it, act on it, and provide top-notch service to Riders. To understand how we built it, start with the foundation: how we measure for real world conditions. PART 1: BUILT FOR REAL WORLD CONDITIONSEvery Signal Is Relative In Smooth Cruiser, context can be boiled down to two simple aspects: the specific stretch of road a Driver is on, and the time the road was driven on. Each behavior is measured against what is typical for that road, at that time of day, day of week, and time of year. This means that a driving event on an icy winter road at night is compared to other winter driving on that road, not to a clear summer afternoon. T

## CaMBRAIN: Real-time, Continuous EEG Inference with Causal State Space Models

DevFeed: [CaMBRAIN: Real-time, Continuous EEG Inference with Causal State Space Models](<https://devfeed.tech/articles/cambrain-real-time-continuous-eeg-inference-with-causal-state-space-models-137258.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.28792>)

Author: Abhilash Durgam, Nyle Siddiqui, Jeffrey A. Chan-Santiago, Qiushi Fu, Elakkat D. Gireesh, Mubarak Shah

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mamba](<https://devfeed.tech/topics/mamba.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [real-time](<https://devfeed.tech/topics/real-time.md>)

Tags: [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [eeg](<https://devfeed.tech/tags/eeg.md>), [inference](<https://devfeed.tech/tags/inference.md>), [mamba](<https://devfeed.tech/tags/mamba.md>), [memory](<https://devfeed.tech/tags/memory.md>), [range](<https://devfeed.tech/tags/range.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [state-space-models](<https://devfeed.tech/tags/state-space-models.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [time-complexity](<https://devfeed.tech/tags/time-complexity.md>), [train](<https://devfeed.tech/tags/train.md>), [training](<https://devfeed.tech/tags/training.md>), [variable](<https://devfeed.tech/tags/variable.md>)

### AI overview

CaMBRAIN is a causal Mamba state space model designed to process variable-length EEG continuously in real time. It addresses the quadratic sequence scaling of attention-based models and the limited context of fixed-length sliding windows, using linear-time inference. Because brief EEG events may be separated by minutes, the authors introduce a multi-stage self-supervised training pipeline intended to teach the model's hidden state to retain long-range context; they argue reconstruction-focused objectives do not do this adequately for streaming models. The abstract reports state-of-the-art results on three EEG datasets and throughput over ten times higher than existing models. These claims are reported in the abstract; detailed methods and evaluation evidence are not included in the supplied excerpt.

### Source excerpt

arXiv:2605.28792v2 Announce Type: replace Abstract: Electroencephalography (EEG) is a critical, non-invasive method to monitor electrical brain activity. EEGs can span anywhere from a couple seconds to multiple hours, posing a major hurdle for existing deep learning methods due to two major factors: (1) existing EEG models are predominantly built upon the attention mechanism, incurring quadratic scaling as the sequence length increases, and (2) raw EEG signals must be processed in a sliding-window fashion due to fixed-length input requirements, preventing global understanding of the entire signal. To this extent, we propose CaMBRAIN - the first Causal, Mamba-based state space model (SSM) capable of real-time inference of EEG signals, arguing that bidirectional approaches are needlessly expensive given the causal, unidirectional nature of EEG. However, training such a model is non-trivial, as crucial EEG events can be extremely brief - within fractions of a second - yet separated by long intervals spanning minutes. Current EEG methods use self-supervised objectives that optimize for signal reconstruction, but these are not well suited for streaming SSMs; they fail to explicitly train the hidden state to retain the salient long-range context needed for streaming inference. We therefore introduce a multi-stage self-supervised training pipeline specifically tailored to encourage long-range memory retention and strong performance on EEG signals, while preserving the linear-time complexity of state space models. CaMBRAIN achieves state-of-the-art (SOTA) results across 3 different EEG datasets with >10x higher throughput than existing models, enabling the first model capable of long-range, continuous inference of variable-length EEG signals.

## Build a real-time next-best-offer recommender using Databricks Feature Views

DevFeed: [Build a real-time next-best-offer recommender using Databricks Feature Views](<https://devfeed.tech/articles/build-a-real-time-next-best-offer-recommender-using-databricks-feature-views-136415.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/build-real-time-next-best-offer-recommender-using-databricks-feature-views>)

Author: Tian Tan; Sixuan He; Nick Joung; Julia Powell

Published: 2026-11-16T17:30:00Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [real-time](<https://devfeed.tech/topics/real-time.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [after](<https://devfeed.tech/tags/after.md>), [aggregate](<https://devfeed.tech/tags/aggregate.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [best](<https://devfeed.tech/tags/best.md>), [build](<https://devfeed.tech/tags/build.md>), [customers](<https://devfeed.tech/tags/customers.md>), [data-science-and-ml](<https://devfeed.tech/tags/data-science-and-ml.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [feature](<https://devfeed.tech/tags/feature.md>), [financial-services](<https://devfeed.tech/tags/financial-services.md>), [next](<https://devfeed.tech/tags/next.md>), [offer](<https://devfeed.tech/tags/offer.md>), [personalization](<https://devfeed.tech/tags/personalization.md>), [platform](<https://devfeed.tech/tags/platform.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [time](<https://devfeed.tech/tags/time.md>), [using](<https://devfeed.tech/tags/using.md>), [want](<https://devfeed.tech/tags/want.md>)

### AI overview

The accelerator shows how Databricks Feature Views can keep training and online serving feature definitions aligned for a retail banking next-best-offer ranker. Its `create_training_set` point-in-time joins reconstruct historical values without future leakage; for inference, Model Serving retrieves customer features from the online store. The notebooks start with batch features and add Kafka clickstream aggregates, avoiding separate hand-written streaming feature logic. In the stated benchmark setup, the ranking endpoint handled about 56 requests per second and stayed within a 50 ms p95 service budget; adding streaming features increased latency by 3-4 ms. Events became model-visible in about 0.1 seconds. Results used a small, untuned configuration, and scale-to-zero can cause cold starts of tens of seconds. The data is synthetic, so the accelerator does not establish revenue or conversion gains, and teams must add eligibility, consent, suitability, and suppression controls before production. Part 2 requires a Unity Catalog Kafka connection and Amazon MSK connectivity.

### Source excerpt

Real-time personalization depends on understanding what customers want in the moment...

## Rebuilding Guardian: How Pinterest Upgraded Its Real-Time Abuse Platform to Modern Infrastructure...

DevFeed: [Rebuilding Guardian: How Pinterest Upgraded Its Real-Time Abuse Platform to Modern Infrastructure...](<https://devfeed.tech/articles/rebuilding-guardian-how-pinterest-upgraded-its-real-time-abuse-platform-to-modern-infrastructure-136560.md>)

Original publisher: [Read original article](<https://medium.com/pinterest-engineering/rebuilding-guardian-how-pinterest-upgraded-its-real-time-abuse-platform-to-modern-infrastructure-56e45ee8f471?source=rss----4c5a5f6279b6---4>)

Author: Pinterest Engineering

Published: 2026-11-16T16:01:03Z

Content type: article

Language: en

Sources: [Pinterest](<https://devfeed.tech/sources/pinterest-engineering-blog-medium.md>)

Topics: [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>)

Tags: [account](<https://devfeed.tech/tags/account.md>), [account-takeover](<https://devfeed.tech/tags/account-takeover.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [blog](<https://devfeed.tech/tags/blog.md>), [cases](<https://devfeed.tech/tags/cases.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [content](<https://devfeed.tech/tags/content.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [data](<https://devfeed.tech/tags/data.md>), [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>), [detection](<https://devfeed.tech/tags/detection.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [flink](<https://devfeed.tech/tags/flink.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [modern](<https://devfeed.tech/tags/modern.md>), [pinterest](<https://devfeed.tech/tags/pinterest.md>), [platform](<https://devfeed.tech/tags/platform.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [starrocks](<https://devfeed.tech/tags/starrocks.md>), [trust-and-safety](<https://devfeed.tech/tags/trust-and-safety.md>)

### AI overview

Guardian helps Pinterest teams analyze real-time event data and turn identified patterns into signals, policies, and downstream actions for use cases such as spam and fraud response, account takeover detection, risk monitoring, and unsafe content enforcement. As Pinterest scaled, Guardian V1's monolithic design made it increasingly difficult to optimize interactive queries, continuous event-to-action processing, and backfills. Guardian V2 was rebuilt on modern data infrastructure to address scale, reliability, and compliance needs while preserving the fast workflow from investigation to operational response. The excerpt does not provide further architectural details.

### Source excerpt

Rebuilding Guardian: How Pinterest Upgraded Its Real-Time Abuse Platform to Modern Infrastructure at Scale Authors: Sharon Xie, Sr. Staff Software Engineer | Zhenxiao Luo, Sr Staff Software Engineer |Yi Pan, Principal Software Engineer | Vikram Deshpande, Sr. Engineering Manager Guardian helps Pinterest teams turn real-time event data into meaningful insights and operational outcomes. It enables subject matter experts to analyze real-time event streams, identify evolving patterns, and operationalize those learnings through signals, policies, and downstream workflows. Over the years, Guardian has supported a wide range of use cases, including spam and fraud response, account takeover detection, risk monitoring, and unsafe content enforcement. As Pinterest scaled, however, the original Guardian platform began to reach its limits. To meet growing demands around scale, reliability, and compliance, we rebuilt the system as Guardian V2 on top of modern data infrastructure. In this blog series, we'll share our journey in building the next generation of Guardian, or Guardian V2. In this first part, we'll discuss the challenges we faced, the architecture shift, and the foundational decisions that improved the platform's reliability, scalability and compliance alignment. Background & Motivation Guardian's biggest strength was the fast loop it enabled from investigation to operational response. In the asynchronous pipeline, teams could analyze enriched event data, identify suspicious patterns, convert those insights into actions, and trigger downstream actions quickly. Preserving that workflow was a core requirement for Guardian V2. As Pinterest scaled, however, the previous generation of Guardian presented several challenges: Scalability, scalability became a growing issue. Guardian V1 used a monolithic architecture to support interactive querying, continuous event to action processing, and backfill workflows, which made it increasingly difficult to optimize each path indepen

## How Infinitas Learning uses ClickHouse Managed Postgres to process billions of events for 1.6M unique learners

DevFeed: [How Infinitas Learning uses ClickHouse Managed Postgres to process billions of events for 1.6M unique learners](<https://devfeed.tech/articles/how-infinitas-learning-uses-clickhouse-managed-postgres-to-process-billions-of-events-for-1-6m-unique-learners-136653.md>)

Original publisher: [Read original article](<https://clickhouse.com/blog/infinitas-learning-clickhouse-managed-postgres>)

Author: ClickHouse

Published: 2026-11-16T10:54:45Z

Content type: news

Language: en

Sources: [ClickHouse Blog](<https://devfeed.tech/sources/clickhouse-blog.md>)

Topics: [clickhouse](<https://devfeed.tech/topics/clickhouse.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [self-hosted databases vs managed services](<https://devfeed.tech/topics/self-hosted-databases-vs-managed-services.md>)

Tags: [1-6](<https://devfeed.tech/tags/1-6.md>), [about](<https://devfeed.tech/tags/about.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [cache](<https://devfeed.tech/tags/cache.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-cache](<https://devfeed.tech/tags/distributed-cache.md>), [events](<https://devfeed.tech/tags/events.md>), [latency](<https://devfeed.tech/tags/latency.md>), [learning](<https://devfeed.tech/tags/learning.md>), [managed](<https://devfeed.tech/tags/managed.md>), [p99](<https://devfeed.tech/tags/p99.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [process](<https://devfeed.tech/tags/process.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [time](<https://devfeed.tech/tags/time.md>), [transactions](<https://devfeed.tech/tags/transactions.md>), [unique](<https://devfeed.tech/tags/unique.md>), [uses](<https://devfeed.tech/tags/uses.md>)

### AI overview

Infinitas Learning separates learner-result transactions from customer-facing analytics: Postgres processes assignment events and updates learner progress, while ClickHouse Cloud serves teacher dashboards. After Neon, the team moved 750 GB to ClickHouse Managed Postgres in 16 hours and completed a no-downtime cutover in 20 minutes; reported P50 transaction latency fell from milliseconds to double-digit microseconds. For analytics, ClickHouse compressed a nearly 2-billion-row table from 300 GB to 27 GB, but cache misses caused P99 spikes up to 13 seconds against a 500 ms target. Self-hosting brought P99 to 300 ms but left the team responsible for operations and running a single replica. With ClickHouse support optimizing data and settings, then enabling distributed cache in Cloud, load-test P99 reached 410 ms. The team plans to connect Postgres and ClickHouse through pg_clickhouse, but that integration is described as a future vision.

### Source excerpt

Infinitas Learning runs transactions for 1.6 million learners on ClickHouse Managed Postgres and serves real-time analytics to teachers from ClickHouse Cloud, where distributed cache cut P99 latency from about 13 seconds to 410 milliseconds.

## How to Run Vision Models on OBS Live Broadcasts

DevFeed: [How to Run Vision Models on OBS Live Broadcasts](<https://devfeed.tech/articles/how-to-run-vision-models-on-obs-live-broadcasts-134502.md>)

Original publisher: [Read original article](<https://roboflow.com/blog/vision-models-on-obs-live-broadcasts>)

Author: Leo Ueno

Published: 2026-11-15T16:10:01Z

Content type: tutorial

Language: en

Sources: [Roboflow](<https://devfeed.tech/sources/roboflow-blog.md>)

Topics: [Computer vision](<https://devfeed.tech/topics/computer-vision.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [broadcast](<https://devfeed.tech/tags/broadcast.md>), [broadcasting](<https://devfeed.tech/tags/broadcasting.md>), [chatgpt](<https://devfeed.tech/tags/chatgpt.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [inference](<https://devfeed.tech/tags/inference.md>), [live](<https://devfeed.tech/tags/live.md>), [live-stream](<https://devfeed.tech/tags/live-stream.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [obs](<https://devfeed.tech/tags/obs.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [roboflow-deploy](<https://devfeed.tech/tags/roboflow-deploy.md>), [run](<https://devfeed.tech/tags/run.md>), [running](<https://devfeed.tech/tags/running.md>), [stream](<https://devfeed.tech/tags/stream.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

OBS broadcasts can be sent to Roboflow Cloud Streaming for live computer vision without a plugin, GPU, or additional hardware. Create a video source to get a server URL and stream key, enter them in OBS under the Custom stream service, and start streaming. Then describe the desired analysis to Roboflow Agent, which builds and deploys a Workflow. In the example, a pretrained model detects people and SAM 3 identifies suits from a text prompt; the Workflow reports live statistics including people per frame, their median share of screen space, and the percentage wearing suits. The excerpt says other encoders that support RTMP or supported protocols can also connect. Treat the stream key like a password.

### Source excerpt

We'll test out running real-time model-inference inference on a OBS live stream, using Roboflow Cloud Streaming.

## What is Real-Time Mode in Apache Spark?

DevFeed: [What is Real-Time Mode in Apache Spark?](<https://devfeed.tech/articles/what-is-real-time-mode-in-apache-spark-136421.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/what-is-real-time-mode>)

Author: Databricks Staff

Published: 2026-11-13T07:49:12Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>)

Tags: [17](<https://devfeed.tech/tags/17.md>), [2026](<https://devfeed.tech/tags/2026.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [autoscaling](<https://devfeed.tech/tags/autoscaling.md>), [availability](<https://devfeed.tech/tags/availability.md>), [batch](<https://devfeed.tech/tags/batch.md>), [batches](<https://devfeed.tech/tags/batches.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [blocking](<https://devfeed.tech/tags/blocking.md>), [data-plus-ai-foundations](<https://devfeed.tech/tags/data-plus-ai-foundations.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [spark](<https://devfeed.tech/tags/spark.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [structured](<https://devfeed.tech/tags/structured.md>), [trigger](<https://devfeed.tech/tags/trigger.md>)

### AI overview

Apache Spark Structured Streaming Real-Time Mode processes records continuously through long-running tasks, reducing latency from seconds to milliseconds without changing the query's core code. It is suited to workloads that need sub-second responses; micro-batch or availableNow costs less when seconds or minutes suffice. On Databricks, classic compute requires Runtime 16.4 LTS or later, Photon, autoscaling and spot instances disabled, plus an enablement setting; serverless use is limited to continuous Lakeflow Spark Declarative Pipelines. Capacity must cover concurrently running tasks. Kafka sources and sinks, foreach sinks, and most operators including stateful ones are supported. Delta output requires Kafka followed by a separate micro-batch query. Monitor Real-Time Mode latency metrics rather than batch duration; the end-to-end metric excludes sink write time. Latency varies with workload and capacity. The text says Databricks has GA support, while open-source availability is rolling out; it also notes stateful support arrives in Spark 4.3.

### Source excerpt

Real-Time Mode is a trigger for Apache Spark™ Structured Streaming that processes...

## Finite-Cadence Refresh Equilibria for Streaming Prior-Anchored Ridge: A Closed-Form Forgetting Window and Spectral Correction

DevFeed: [Finite-Cadence Refresh Equilibria for Streaming Prior-Anchored Ridge: A Closed-Form Forgetting Window and Spectral Correction](<https://devfeed.tech/articles/finite-cadence-refresh-equilibria-for-streaming-prior-anchored-ridge-a-closed-form-forgetting-window-and-spectral-correction-132025.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14586>)

Author: Mrigank Shekhar Singh

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [cadence](<https://devfeed.tech/tags/cadence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-gt](<https://devfeed.tech/tags/cs-gt.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [electricity](<https://devfeed.tech/tags/electricity.md>), [formalized](<https://devfeed.tech/tags/formalized.md>), [lean](<https://devfeed.tech/tags/lean.md>), [lean-4](<https://devfeed.tech/tags/lean-4.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [random](<https://devfeed.tech/tags/random.md>), [regression](<https://devfeed.tech/tags/regression.md>), [risk](<https://devfeed.tech/tags/risk.md>), [spar](<https://devfeed.tech/tags/spar.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [window](<https://devfeed.tech/tags/window.md>)

### AI overview

For prior-anchored ridge regression on drifting streams, refreshing the anchor every k steps yields a finite-cadence equilibrium: each spectral direction trades lag against reduced estimation variance. The abstract gives a cycle-averaged risk optimum for the forgetting window and a plug-in formula for anchor strength. A finite-dimensional random-matrix correction renormalizes the anchor and reportedly matches simulations within 5% at fixed operating points and 13% at the method's own operating point; rank-deficient populations reduce to their active subspace. SPAR calibrates window and strength online from residuals. In reported evaluations, its window responds to drift without divergence on a 27-configuration grid, and grid defaults transfer to an hourly electricity stream and BCI IV 2b. The core algebra, refresh chain and regret bounds are formalized in Lean 4.

### Source excerpt

arXiv:2611.14586v1 Announce Type: cross Abstract: Recursive estimators on drifting streams must forget stale data fast enough to track the target, yet slowly enough to average out noise. In prior-anchored ridge regression, this trade-off admits a closed-form spectral decomposition in which each eigen-direction interpolates between the data solution and the prior with pull weight $s_i = \rho / (\mu_i + \rho)$. When the anchor is refreshed every $k$ steps, we show that the moving-frame offset converges to a finite-cadence equilibrium that lags the pure-leaky path by an explicit $\Delta_k(s)$ while reducing estimation variance by a factor $\Psi(s, r_k)$. Minimizing the cycle-averaged risk gives $\omega^* = (4\nu^2 \Sigma / c')^{1/3}$, $c' = \sigma^2 d_{\mathrm{eff}}(\rho)$, and the plug-in strength $\rho^* = \sigma^2 / (n\delta^2)$. A random-matrix deterministic equivalent extends both laws to finite dimension, renormalizing the anchor to $\rho_{\mathrm{eff}} = \rho (e^{\zeta q} - 1) / (\zeta q)$; it matches simulation within 5% at fixed operating points and within 13% at the method's own, and rank-deficient populations reduce exactly to their active subspace. The resulting method, SPAR, calibrates window and strength online from residuals. On a 27-configuration grid the calibrated window opens on drift without divergence, and the grid defaults transfer to an hourly electricity stream and to BCI IV 2b. The core algebra, refresh chain, and regret bounds are formalized in Lean 4.

## UniPipe: Unified Inference Serving and Fine-Tuning for Deadline-Constrained DNN Surrogate Workflows

DevFeed: [UniPipe: Unified Inference Serving and Fine-Tuning for Deadline-Constrained DNN Surrogate Workflows](<https://devfeed.tech/articles/unipipe-unified-inference-serving-and-fine-tuning-for-deadline-constrained-dnn-surrogate-workflows-132910.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14748>)

Author: Jaiaid Mobin, Avinash Maurya, M. Mustafa Rafique, Bogdan Nicolae

Published: 2026-11-13T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Advanced AI workflows](<https://devfeed.tech/topics/advanced-ai-workflows.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>)

Tags: [batch](<https://devfeed.tech/tags/batch.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [cs-sy](<https://devfeed.tech/tags/cs-sy.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [data](<https://devfeed.tech/tags/data.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [dnn](<https://devfeed.tech/tags/dnn.md>), [edge](<https://devfeed.tech/tags/edge.md>), [eess-sy](<https://devfeed.tech/tags/eess-sy.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [imaging](<https://devfeed.tech/tags/imaging.md>), [inference](<https://devfeed.tech/tags/inference.md>), [latency](<https://devfeed.tech/tags/latency.md>), [linear](<https://devfeed.tech/tags/linear.md>), [offline](<https://devfeed.tech/tags/offline.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [sensor](<https://devfeed.tech/tags/sensor.md>), [serving](<https://devfeed.tech/tags/serving.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

UniPipe combines surrogate inference and fine-tuning in one GPU context to reduce latency variability and deadline misses in streaming scientific workflows. It merges training and inference samples in a shared forward pass, applies backpropagation only to training samples, and uses an offline polynomial-time scheduler to choose batch composition based on nonlinear GPU latency and recurring workload intervals. In a Ptychographic reconstruction evaluation with sensor rates up to 5,000 patterns/sec, it reduced reconstruction MSE by up to 30% and inference MSE by up to 50% versus a pretrained baseline. Compared with a multi-context baseline, it cut deadline misses by 13-22%, narrowed latency spread from 106% to 57% of mean latency, and increased total throughput. These findings are from the abstract; further evaluation details are not provided.

### Source excerpt

arXiv:2611.14748v1 Announce Type: new Abstract: DNN surrogates are increasingly used to accelerate scientific workflows that must process high-rate sensor streams in near real time, exemplified by Ptychographic imaging at next-generation X-ray light sources. Such surrogates deliver 100-1000x higher throughput than iterative solvers, enabling faster steering of experiments. To preserve surrogate accuracy during an experiment, as the input distribution evolves, state-of-the-art deployments freeze the pretrained surrogate or run a separate CUDA context that fine-tunes concurrently with serving. However, multi-context concurrency causes significant contention on the GPU, inflating both the mean and the variance of inference latency, triggering deadline misses in workflows where sensor data has to be processed within a hard per-unit deadline. To address this challenge, we propose UniPipe, a single-context system that unifies inference serving and fine-tuning of a DNN surrogate on a single edge GPU. UniPipe is driven by three key insights: both serving and fine-tuning require a forward pass and can be fused into one; the per-sample latency of modern GPUs is non-linear in batch size; and the workload is repetitive across intervals, enabling offline scheduling. Specifically, UniPipe fuses training and inference data in a single forward pass with a selective backward pass on the training subset, and uses a polynomial-time scheduler that exploits the latency curve to compute the batch composition. We evaluate UniPipe on a real-world Ptychographic reconstruction workflow with sensor rates up to 5000 patterns/sec.; UniPipe lowers reconstruction MSE by up to 30% and inference MSE by up to 50% over a pretrained baseline, and reduces the deadline-miss rate by 13%-22% over a multi-context baseline. UniPipe's unified deployment reduces the latency spread from 106% to 57% of the mean, and increases total throughput compared to multi-context baseline.

## Lakehouse//RT: Scale real-time analytics, not your bill

DevFeed: [Lakehouse//RT: Scale real-time analytics, not your bill](<https://devfeed.tech/articles/lakehouse-rt-scale-real-time-analytics-not-your-bill-130981.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/lakehousert-scale-real-time-analytics-not-your-bill>)

Author: JooHo Yeo; Kaitlin Baumgardner

Published: 2026-11-12T16:53:43Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [bill](<https://devfeed.tech/tags/bill.md>), [compute](<https://devfeed.tech/tags/compute.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [data-warehousing](<https://devfeed.tech/tags/data-warehousing.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [platform](<https://devfeed.tech/tags/platform.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-time-analytics](<https://devfeed.tech/tags/real-time-analytics.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

Lakehouse//RT aims to handle high, bursty real-time query volumes without scaling costs proportionally. Its approach combines higher query concurrency and faster end-to-end execution with capacity scaling in small increments, including one node at a time. It queries governed lakehouse data directly, avoiding a separate serving stack and duplicated data. The text reports Aurasell sustained over 250 QPS on one small warehouse, with queries 4-5x faster and 10x lower infrastructure costs; VisionHeight reported over 90% lower costs and nearly 3x faster responses than Apache Pinot on its heaviest query; nOps reported infrastructure costs cut by more than 6x and eliminated 40TB of duplicated data. These are vendor-reported customer outcomes. Lakehouse//RT is described as available in beta.

### Source excerpt

AI agents are pushing the limits of data warehouses. A human analyst might run a...

## Another Way Police Are Locating People from Their Smartphones

DevFeed: [Another Way Police Are Locating People from Their Smartphones](<https://devfeed.tech/articles/another-way-police-are-locating-people-from-their-smartphones-130606.md>)

Original publisher: [Read original article](<https://www.schneier.com/blog/archives/2026/11/another-way-police-are-locating-people-from-their-smartphones.html>)

Author: Bruce Schneier

Published: 2026-11-12T12:02:58Z

Content type: news

Language: en

Sources: [Schneier on Security](<https://devfeed.tech/sources/schneier-on-security.md>)

Topics: [data](<https://devfeed.tech/topics/data.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [all](<https://devfeed.tech/tags/all.md>), [application](<https://devfeed.tech/tags/application.md>), [apps](<https://devfeed.tech/tags/apps.md>), [bloomberg](<https://devfeed.tech/tags/bloomberg.md>), [build](<https://devfeed.tech/tags/build.md>), [can](<https://devfeed.tech/tags/can.md>), [company](<https://devfeed.tech/tags/company.md>), [data](<https://devfeed.tech/tags/data.md>), [data-collection](<https://devfeed.tech/tags/data-collection.md>), [email](<https://devfeed.tech/tags/email.md>), [gaming](<https://devfeed.tech/tags/gaming.md>), [geolocation](<https://devfeed.tech/tags/geolocation.md>), [information](<https://devfeed.tech/tags/information.md>), [it-s](<https://devfeed.tech/tags/it-s.md>), [law](<https://devfeed.tech/tags/law.md>), [law-enforcement](<https://devfeed.tech/tags/law-enforcement.md>), [location](<https://devfeed.tech/tags/location.md>), [look](<https://devfeed.tech/tags/look.md>), [map](<https://devfeed.tech/tags/map.md>), [people](<https://devfeed.tech/tags/people.md>), [police](<https://devfeed.tech/tags/police.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [smartphones](<https://devfeed.tech/tags/smartphones.md>), [surveillance](<https://devfeed.tech/tags/surveillance.md>), [uncategorized](<https://devfeed.tech/tags/uncategorized.md>)

### AI overview

Penlink's Webloc helps police locate smartphones by collecting location data from commonly used weather, gaming, email, and religious apps. It can show a phone's movements over the previous 12 months, refresh data hourly, and let officers view phones present at a specified place and time. Company documents reviewed by Bloomberg describe analysis of billions of location data points. The supplied excerpt does not explain how the system obtains the app data or address its limitations.

### Source excerpt

Penlink's Webloc is an application that locates people using information from the weather, gaming, email, and religious apps on their phone. It's sold to the police. The tool collects vast quantities of location data from smartphones via commonly used weather, gaming, email and religious apps. It can build a detailed picture of where a smartphone has been over the past 12 months, and update the data hourly, showing the movements of people on a near real-time basis. Law enforcement officers can use Webloc to zero in on a map and look at all phones that were present at a specific place and time, analyzing "billions of location data points," according to company documents reviewed by Bloomberg...

[Next page](<https://devfeed.tech/topics/streaming-data-analytics.md?cursor=WyIyMDI2LTExLTEyVDEyOjAyOjU4KzAwOjAwIiwgIjBkOGUzM2RhLWM1N2YtNDZlNS1iM2JkLTY0NjE5ZmE1NzNkMCJd>)