# Event-Streaming

Event streaming is the continuous capture, storage, processing, and routing of data events in real time.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Doing More by Storing Less: Speeding Up Real-Time Machine Learning

DevFeed: [Doing More by Storing Less: Speeding Up Real-Time Machine Learning](<https://devfeed.tech/articles/doing-more-by-storing-less-speeding-up-real-time-machine-learning-144308.md>)

Original publisher: [Read original article](<https://medium.com/feedzaitech/doing-more-by-storing-less-speeding-up-real-time-machine-learning-23639ae00e22?source=rss----e11168e7fe6b---4>)

Author: Augusto Peres

Published: 2026-11-19T15:40:42Z

Content type: article

Language: en

Sources: [Feedzai](<https://devfeed.tech/sources/feedzai.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>)

Tags: [aggregate](<https://devfeed.tech/tags/aggregate.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [counts](<https://devfeed.tech/tags/counts.md>), [database](<https://devfeed.tech/tags/database.md>), [detection](<https://devfeed.tech/tags/detection.md>), [event](<https://devfeed.tech/tags/event.md>), [financial-applications](<https://devfeed.tech/tags/financial-applications.md>), [fraud-detection](<https://devfeed.tech/tags/fraud-detection.md>), [heavy-hitter](<https://devfeed.tech/tags/heavy-hitter.md>), [inference](<https://devfeed.tech/tags/inference.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [latency](<https://devfeed.tech/tags/latency.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

Feedzai's event filtering reduces costly read-modify-write database cycles for high-volume entities while aiming to preserve useful feature profiles. Below a chosen per-entity write-rate threshold, every event is stored; above it, events are randomly included with probability set by the threshold divided by the estimated arrival rate. Kept events are weighted by the inverse of that probability to compensate counts and sums for discarded updates. The system can estimate arrival rates from filtered persisted statistics, updating and scaling those estimates through the same sampling process rather than tracking every entity's rate in memory; the article describes this as self-correcting. It says the approach fits power-law transaction volumes and reports that model accuracy did not decline under heavy filtering, with small gains in some cases attributed to training noise. The excerpt provides no quantitative performance results.

### Source excerpt

Modern fraud detection systems rely heavily on tracking behavior over time. For example: How much money has a person spent this month? Or, how many purchases have they made in the last week? Any event-streaming system worth its salt tracks hundreds of such metrics. Every time a new transaction even arrives, these are retrieved from a database, updated, and then written back to persistent storage. This process is known as the read-modify-write cycle, and its final write stage is a major source of latency, database contention, and infrastructure cost. But here is the catch: In many financial applications, a small set of heavy-hitter entities triggers most of these cycles and accounts for a disproportionate number of transactions. Statistically, the more events we process for any entity, the less information each individual event adds to their overall profile. Thus, here at Feedzai we developed a process and system that automatically stops writing every single event for large financial entities, saving significant compute without losing downstream ML accuracy. A friendly introduction to feature profiles Before we go any deeper into discarding events, we briefly explain what feature profiles are. Essentially, feature profiles aggregate events over a period of time. Two common examples are: Counts: How many times have we seen an entity doing something? e.g., the number of transactions for a specific debit card in the last month. Sums: The running total of a quantity associated with an event, e.g., the total spend of a user in the last week. In a real-time fraud detection system, when a single event arrives, the feature profiles for its associated entity must be retrieved from persistent storage, (e.g., a relational database or key-value store). Then, the profiles must be updated with the information on the new event. Next, the updated profiles are sent to a Machine Learning inference service for classification (e.g., fraud or not fraud) and written back to persistent sto

## Aderant modernizes analytics data pipeline with Apache Iceberg and AWS Lambda

DevFeed: [Aderant modernizes analytics data pipeline with Apache Iceberg and AWS Lambda](<https://devfeed.tech/articles/aderant-modernizes-analytics-data-pipeline-with-apache-iceberg-and-aws-lambda-141916.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/aderant-modernizes-analytics-data-pipeline-with-apache-iceberg-and-aws-lambda/>)

Author: Sham Birbahadur

Published: 2026-11-19T00:36:23Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [event-driven data processing](<https://devfeed.tech/topics/event-driven-data-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [AWS Database Migration Service](<https://devfeed.tech/topics/aws-database-migration-service.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [acid](<https://devfeed.tech/tags/acid.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [batch](<https://devfeed.tech/tags/batch.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cost](<https://devfeed.tech/tags/cost.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [faster](<https://devfeed.tech/tags/faster.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [legal](<https://devfeed.tech/tags/legal.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [queries](<https://devfeed.tech/tags/queries.md>)

### AI overview

Aderant replaced Stridyn's 15-minute batch pipeline with an event-driven medallion architecture that processes CDC changes from 700 client environments in single-digit seconds to bronze-layer Apache Iceberg tables. AWS DMS delivers Parquet files to S3; S3 notifications and SQS invoke Lambda for ingestion, while scheduled AWS Glue jobs transform data through silver and gold layers. Incremental Glue jobs handle 77 standard gold tables; 34 complex tables use Iceberg materialized views and snapshot comparisons to send changed rows to native Amazon Redshift tables. Aderant reports 10-26x faster queries on native Redshift tables, under-100-millisecond dashboard queries, and 77 percent lower pipeline compute costs versus streaming alternatives. The account says this approach supports schema changes and concurrent tenant workloads; it also describes a possible future Kinesis path for high-volume tenants.

### Source excerpt

Learn how Aderant modernized the data pipeline behind its Stridyn legal analytics platform with Apache Iceberg, AWS Lambda, and AWS Glue, moving from 15-minute batch cycles to an event-driven medallion architecture that processes change data capture events in single-digit seconds for 700 tenants, with 10-26x faster queries and 77 percent lower cost.

## Wait for pull request events with GitHub CLI

DevFeed: [Wait for pull request events with GitHub CLI](<https://devfeed.tech/articles/wait-for-pull-request-events-with-github-cli-141432.md>)

Original publisher: [Read original article](<https://github.blog/changelog/2026-11-18-wait-for-pull-request-events-with-github-cli>)

Author: Allison

Published: 2026-11-19T00:00:13Z

Content type: release

Language: en

Sources: [GitHub Changelog](<https://devfeed.tech/sources/github-changelog.md>)

Topics: [GitHub](<https://devfeed.tech/topics/github.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [GitHub API](<https://devfeed.tech/topics/github-api.md>)

Tags: [cli](<https://devfeed.tech/tags/cli.md>), [client-apps](<https://devfeed.tech/tags/client-apps.md>), [ecosystem-accessibility](<https://devfeed.tech/tags/ecosystem-accessibility.md>), [events](<https://devfeed.tech/tags/events.md>), [github](<https://devfeed.tech/tags/github.md>), [github-cli](<https://devfeed.tech/tags/github-cli.md>), [pull-request](<https://devfeed.tech/tags/pull-request.md>), [release](<https://devfeed.tech/tags/release.md>)

### AI overview

With `gh` v2.105.0 or later, `gh event listen` can wait for selected pull request events and print each as a line of JSON, replacing polling loops. It exits after the first event by default; `--count`, `--follow`, and `--timeout` control how long it listens, while `--jq` and `--template` select fields. Supported events include submitted reviews, pull request state or head changes, merge conflicts, and completed check suites. Use `gh event subscription create`, `list`, `view`, and `delete` to manage subscriptions directly. The commands are in preview and require access to Event Streams, also in preview on GitHub.com; access is available through its waitlist.

### Source excerpt

GitHub CLI now supports GitHub Event Streams. With gh v2.105.0, you and your agents can wait for a pull request to change with a single command instead of polling it... The post Wait for pull request events with GitHub CLI appeared first on The GitHub Blog.

## How United Airlines streams database changes to Apache Iceberg with Amazon Data Firehose

DevFeed: [How United Airlines streams database changes to Apache Iceberg with Amazon Data Firehose](<https://devfeed.tech/articles/how-united-airlines-streams-database-changes-to-apache-iceberg-with-amazon-data-firehose-141918.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/how-united-airlines-streams-database-changes-to-apache-iceberg-with-amazon-data-firehose/>)

Author: Guillaume Dupont

Published: 2026-11-18T20:37:39Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Amazon Aurora](<https://devfeed.tech/topics/amazon-aurora.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Visual Studio Code](<https://devfeed.tech/topics/visual-studio-code.md>)

Tags: [airlines](<https://devfeed.tech/tags/airlines.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [amazon-data-firehose](<https://devfeed.tech/tags/amazon-data-firehose.md>), [amazon-kinesis](<https://devfeed.tech/tags/amazon-kinesis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cloudformation](<https://devfeed.tech/tags/aws-cloudformation.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [dms](<https://devfeed.tech/tags/dms.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [kinesis-data-streams](<https://devfeed.tech/tags/kinesis-data-streams.md>), [streams](<https://devfeed.tech/tags/streams.md>), [united-airlines](<https://devfeed.tech/tags/united-airlines.md>)

### AI overview

United Airlines describes a near-real-time CDC pipeline that moves Aurora MySQL changes through AWS DMS, Kinesis Data Streams, Lambda, and Amazon Data Firehose into Apache Iceberg tables on S3. Firehose applies inserts, updates, and deletes as Iceberg snapshots; the open tables can be queried through AWS Glue, Redshift, or Athena. United reports 5x lower average query latency and 2x shorter downstream Glue job processing time in its testing, with a low-volume test reaching about 10 seconds end-to-end. Buffer settings trade latency against cost and file size. The walkthrough uses DMS Serverless, Athena for validation, and a simplified single-table CloudFormation template; United's production setup uses provisioned DMS and primarily Redshift Serverless. The template's shard count and other demo settings are not production-tuned, and schema evolution and multi-table replication require additional handling.

### Source excerpt

United Airlines built a near-real-time change data capture pipeline that replicates reference data from Amazon Aurora MySQL to open Apache Iceberg tables using AWS DMS, Amazon Kinesis Data Streams, and Amazon Data Firehose. This post shows how to reproduce the serverless, multi-engine analytics pattern with AWS CloudFormation.

## Time-Series Models in the Streamhouse: A New Frontier and Use Cases

DevFeed: [Time-Series Models in the Streamhouse: A New Frontier and Use Cases](<https://devfeed.tech/articles/time-series-models-in-the-streamhouse-a-new-frontier-and-use-cases-143009.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/ibm-research/time-series-powered-streamhouse>)

Author: Nicholas Fuller; Sean Falconer; Ayhan Sebin; Pankaj Dayama; Roman Vaculin; Bilge Zeren Aksu; Julian Payne; Mountu Jinwala; Wesley M Gifford; Chandra Reddy; Vijay Ekambaram; Arindam Jati; Robert Metzge

Published: 2026-11-18T15:00:23Z

Content type: release

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Time Series](<https://devfeed.tech/topics/time-series.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Confluent Platform](<https://devfeed.tech/topics/confluent-platform.md>)

Tags: [anomaly-detection](<https://devfeed.tech/tags/anomaly-detection.md>), [business](<https://devfeed.tech/tags/business.md>), [can](<https://devfeed.tech/tags/can.md>), [cases](<https://devfeed.tech/tags/cases.md>), [change](<https://devfeed.tech/tags/change.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [confluent](<https://devfeed.tech/tags/confluent.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cpus](<https://devfeed.tech/tags/cpus.md>), [detection](<https://devfeed.tech/tags/detection.md>), [downloads](<https://devfeed.tech/tags/downloads.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [form](<https://devfeed.tech/tags/form.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [help](<https://devfeed.tech/tags/help.md>), [models](<https://devfeed.tech/tags/models.md>), [series](<https://devfeed.tech/tags/series.md>), [time](<https://devfeed.tech/tags/time.md>), [time-series](<https://devfeed.tech/tags/time-series.md>), [use-cases](<https://devfeed.tech/tags/use-cases.md>)

### AI overview

IBM time-series foundation models are available as native Confluent functions in managed Confluent Cloud and on-premises Confluent Platform for Apache Flink, running on CPUs. Flink can apply forecasting, anomaly detection, and time-series representations to streaming data with SQL, putting results alongside current events. Pretrained models offer a starting point across many series; workflows can incorporate recent observations, business rules, and human review. Examples include forecasting production quality to evaluate setpoints, flagging unusual store sales against a chosen confidence level, refreshing store-level demand forecasts to inform reorder points, adding individual spending patterns to existing payment checks, and pacing trades from short-term volume forecasts. Similarity search with TSPulse requires a separately built embedding pipeline and index, and can help investigate patterns but cannot establish root cause. These scenarios are illustrative; outcomes depend on model requirements, pipeline configuration, and application logic.

### Source excerpt

**Real-time forecasting, anomaly detection, and optimization, powered by IBM time-series models with more than 50 million downloads. Now available natively on Confluent, in the cloud and on premises, running on CPUs.** Every business generates a continuous record of change: a machine reading, a payment, a sale, an order, or a change in inventory. These observations form time series, and they carry signals that can help teams anticipate what happens next.

## Is your observability pipeline becoming the bottleneck?

DevFeed: [Is your observability pipeline becoming the bottleneck?](<https://devfeed.tech/articles/is-your-observability-pipeline-becoming-the-bottleneck-140944.md>)

Original publisher: [Read original article](<https://www.confluent.io/blog/is-your-observability-pipeline-becoming-the-bottleneck/>)

Author: Confluent Staff

Published: 2026-11-18T05:04:51Z

Content type: article

Language: en

Sources: [Confluent: Data in motion](<https://devfeed.tech/sources/confluent-data-in-motion.md>)

Topics: [observability pipelines](<https://devfeed.tech/topics/observability-pipelines.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [scalable-architecture](<https://devfeed.tech/topics/scalable-architecture.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [data-streaming](<https://devfeed.tech/tags/data-streaming.md>), [guide](<https://devfeed.tech/tags/guide.md>), [limits](<https://devfeed.tech/tags/limits.md>), [observability](<https://devfeed.tech/tags/observability.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>)

### AI overview

Telemetry spikes can expose observability pipeline limits: downstream systems may delay or drop events just when engineers need them, while rising volumes can increase ingestion costs, complicate adding destinations, and leave engineers maintaining scripts and integrations. The excerpt identifies these four patterns as signs to reassess filtering, processing, and data flow. A streaming layer between collection and destinations may help in some environments, but adds operational responsibilities and cost; whether it is worthwhile depends on the benefits and the technical requirements of the specific environment. The referenced guide explores those requirements and where IBM Confluent Data Streaming Platform may fit.

### Source excerpt

Discover four signs your observability pipeline is reaching its limits and learn how data streaming can help you build a more scalable architecture. Get the guide.

## Gateway-Free Kafka Streaming from FPGA-Based Detectors: A Bare-Metal Pipelined Producer on Zynq-7000

DevFeed: [Gateway-Free Kafka Streaming from FPGA-Based Detectors: A Bare-Metal Pipelined Producer on Zynq-7000](<https://devfeed.tech/articles/gateway-free-kafka-streaming-from-fpga-based-detectors-a-bare-metal-pipelined-producer-on-zynq-7000-140403.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22674>)

Author: Fernanda Zapata Bascu\\~n\\'an, Mat\\'ias Hampel

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Networking](<https://devfeed.tech/sources/arxiv-networking.md>)

Topics: [Xilinx Zynq](<https://devfeed.tech/topics/xilinx-zynq.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [fpga](<https://devfeed.tech/topics/fpga.md>), [lwIP](<https://devfeed.tech/topics/lwip.md>), [IO](<https://devfeed.tech/topics/io.md>)

Tags: [bare-metal](<https://devfeed.tech/tags/bare-metal.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-ni](<https://devfeed.tech/tags/cs-ni.md>), [dma](<https://devfeed.tech/tags/dma.md>), [errors](<https://devfeed.tech/tags/errors.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [fpga](<https://devfeed.tech/tags/fpga.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [lwip](<https://devfeed.tech/tags/lwip.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [topic](<https://devfeed.tech/tags/topic.md>)

### AI overview

A bare-metal Kafka producer on a Zynq-7000 streams detector events directly from programmable logic to a standard broker, without an intermediate gateway. Events move over AXI4-Stream and AXI DMA to the processing system, which builds ProduceRequest messages and sends them through lwIP's raw TCP/IP API. A configurable limit on outstanding requests was tested at pipeline depths 1, 2, 4, and 8 using two DMA sources across 80 runs totaling 400,000 events. Mean throughput rose from about 5.3 kevents/s at depth 1 to 16.2 kevents/s at depth 4 and 16.8-17.5 kevents/s at depth 8. Median publish-to-acknowledgement time was lowest at depth 4 (0.66 ms), versus 0.83-0.86 ms at depth 8; depth 4's P95 was 0.81-0.83 ms. All runs had the expected frame sequence in exported metrics and no recorded Kafka errors. The excerpt reports these measurements but does not establish performance beyond the tested setup.

### Source excerpt

arXiv:2611.22674v1 Announce Type: new Abstract: We present a bare-metal Apache Kafka producer implemented on a Zynq-7000 System-on-Chip (SoC) for event-driven data acquisition. Events are generated in Programmable Logic (PL), transferred to the Processing System (PS) through AXI4-Stream and AXI DMA, and published to a standard Kafka broker without an intermediate gateway computer. The PS constructs ProduceRequest messages and transmits them through the lwIP raw TCP/IP API. A configurable pipeline limits the number of outstanding Kafka requests. Pipeline depths of D = 1, 2, 4, and 8 were evaluated with two independent DMA sources. For each depth, 20 independent runs of 5,000 events were acquired, for 400,000 event records in total. Each event carries a 32-bit frame counter and a 32-bit PL timestamp. PS-side instrumentation records DMA start and completion, publication, successful local tcp_write() acceptance, and reception of the corresponding Kafka acknowledgement. Measurement records are exported to a separate Kafka topic after the timed event stream has completed. Each of the 80 runs contained 5,000 records with frame values from 0 to 4,999, with no sequence discontinuities in the exported metric records and no recorded Kafka errors. Mean operational throughput was approximately 5.3 kevents/s at D = 1, 9.4-9.6 kevents/s at D = 2, 16.2 kevents/s at D = 4, and 16.8-17.5 kevents/s at D = 8. The median publish-to-acknowledgement interval was 1.62-1.64 ms at D = 1, 0.98 ms at D = 2, 0.66 ms at D = 4, and 0.83-0.86 ms at D = 8. At D = 4, the corresponding P95 values were 0.81-0.83 ms.

## CheckCo: Checkpoint-Coordinated Load Balancingfor Skewed Event Streams at Scale

DevFeed: [CheckCo: Checkpoint-Coordinated Load Balancingfor Skewed Event Streams at Scale](<https://devfeed.tech/articles/checkco-checkpoint-coordinated-load-balancingfor-skewed-event-streams-at-scale-140388.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22476>)

Author: Muhammad Anis Uddin Nasir, Reza Sadraei

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [checkpoint](<https://devfeed.tech/tags/checkpoint.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [event-streams](<https://devfeed.tech/tags/event-streams.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>), [routing](<https://devfeed.tech/tags/routing.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [workers](<https://devfeed.tech/tags/workers.md>)

### AI overview

CheckCo uses checkpoint barriers to adapt routing to skewed event streams without prior key-frequency statistics. Workers count events per key locally; at each barrier, counts are merged to split keys roughly in proportion to their estimated frequency and pack those splits onto workers for the next checkpoint window. The authors derive bounds on load imbalance and total key fan-out. On a production pipeline handling over 100 billion events daily, the most-loaded worker received 9.8% more events than average over a day (imbalance ratio 1.10), compared with 8.03 for key-based routing in simulation. In simulation, CheckCo used less than half the fan-out of earlier partitioners with comparable balance. Routing reflects distribution shifts from the next checkpoint, and the authors report balanced load through job restarts.

### Source excerpt

arXiv:2611.22476v1 Announce Type: new Abstract: Event streams are rarely uniform: a few dominant keys can generate orders of magnitude more events than the rest, overloading the workers that receive them and capping throughput. We present CheckCo, a checkpoint-coordinated load-balancing algorithm built on a dynamic oracle that estimates per-key frequencies from the live stream, with no prior statistics. Workers count events per key locally; at each checkpoint barrier, the counts are merged into a partitioning function that divides each key into a number of splits roughly proportional to its estimated frequency and packs the splits onto workers, and this function routes the next checkpoint window. We derive approximation bounds on the resulting load imbalance from classical scheduling results and bound its total key fan-out. On a production pipeline processing over 100 billion events per day, the most loaded worker receives only 9.8% more events than the average over a day (imbalance ratio 1.10), against 8.03 for key-based routing in simulation; in simulation, CheckCo also needs less than half the fan-out of earlier partitioners that reach comparable balance. Shifts in the key distribution are reflected in routing from the next checkpoint on, and the load stays balanced through job restarts.

## Get real-time updates on your data pipelines with Fivetran webhooks | Blog | Fivetran

DevFeed: [Get real-time updates on your data pipelines with Fivetran webhooks | Blog | Fivetran](<https://devfeed.tech/articles/get-real-time-updates-on-your-data-pipelines-with-fivetran-webhooks-blog-fivetran-140234.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/get-real-time-updates-on-your-data-pipelines-with-fivetran-webhooks>)

Author: Jimmy Hooker

Published: 2026-11-18T03:12:14Z

Content type: release

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [real-time](<https://devfeed.tech/topics/real-time.md>)

Tags: [alerts](<https://devfeed.tech/tags/alerts.md>), [api](<https://devfeed.tech/tags/api.md>), [code](<https://devfeed.tech/tags/code.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [data](<https://devfeed.tech/tags/data.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [devops](<https://devfeed.tech/tags/devops.md>), [devops-tools](<https://devfeed.tech/tags/devops-tools.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [end-user](<https://devfeed.tech/tags/end-user.md>), [events](<https://devfeed.tech/tags/events.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [updates](<https://devfeed.tech/tags/updates.md>), [webhooks](<https://devfeed.tech/tags/webhooks.md>)

### AI overview

Fivetran webhooks push updates to a configured endpoint when selected events occur, replacing repeated API polling or workflows built from platform connector data. The excerpt lists beta events for sync start and end, connector status, transformation runs, and dbt runs, each with start, success, or failure events where applicable. Suggested uses include notifying users when synced data is ready, triggering orchestration, and alerting PagerDuty or other DevOps tools when a pipeline breaks. The article points to an Express and Ngrok sample project for setup; the excerpt does not detail configuration requirements or limitations.

### Source excerpt

With Fivetran webhooks, developers can use real-time messages to power user experiences, transform data, drive error alerting and more.

## Reduce the complexity and cost of real-time processing with queues in Amazon Kinesis Data Streams

DevFeed: [Reduce the complexity and cost of real-time processing with queues in Amazon Kinesis Data Streams](<https://devfeed.tech/articles/reduce-the-complexity-and-cost-of-real-time-processing-with-queues-in-amazon-kinesis-data-streams-140167.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/reduce-the-complexity-and-cost-of-real-time-processing-with-queues-in-amazon-kinesis-data-streams/>)

Author: Dirk Fröhner

Published: 2026-11-17T23:17:16Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-kinesis](<https://devfeed.tech/tags/amazon-kinesis.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [backlog](<https://devfeed.tech/tags/backlog.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [demand](<https://devfeed.tech/tags/demand.md>), [instead](<https://devfeed.tech/tags/instead.md>), [kinesis](<https://devfeed.tech/tags/kinesis.md>), [kinesis-data-streams](<https://devfeed.tech/tags/kinesis-data-streams.md>), [offers](<https://devfeed.tech/tags/offers.md>), [one](<https://devfeed.tech/tags/one.md>), [processing](<https://devfeed.tech/tags/processing.md>), [queues](<https://devfeed.tech/tags/queues.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [streams](<https://devfeed.tech/tags/streams.md>), [time](<https://devfeed.tech/tags/time.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Amazon Kinesis Data Streams queues provide per-record delivery tracking for applications that can process records independently. Consumers receive batches of up to 10 records, process them, and acknowledge them; unacknowledged records become available again after a configurable visibility timeout. Delivery is at least once and ordering is best effort, so processing should be idempotent. Records that exceed the default six receives without acknowledgment go to a dead-letter state, where operators can inspect, restore, or discard them. Queues scale consumer instances with backlog rather than shard count and remove the need for a KCL lease table and per-shard custom metrics. Multiple queues, stream consumers, and managed delivery can use the same stream without a relay or duplicate ingest. Use direct stream consumers when processing depends on record order or prior state; use queues for independent work. AWS reports up to 43% lower processing cost, but its synthetic benchmark used an I/O-bound handler; savings depend on workload and configuration, and CPU-bound work narrows the compute benefit.

### Source excerpt

Amazon Kinesis Data Streams now offers queues, a new way to consume the records of an on-demand stream. A queue delivers records one by one, tracks each record's delivery state, redelivers unacknowledged records, sets persistently failing records aside as dead letters, and scales with your backlog instead of your shard count.

## Modernizing ACH: Why the Workhorse Rail Needs a New Engine

DevFeed: [Modernizing ACH: Why the Workhorse Rail Needs a New Engine](<https://devfeed.tech/articles/modernizing-ach-why-the-workhorse-rail-needs-a-new-engine-136911.md>)

Original publisher: [Read original article](<https://hackernoon.com/modernizing-ach-why-the-workhorse-rail-needs-a-new-engine?source=rss>)

Author: Sanya Kapoor

Published: 2026-11-17T08:14:49Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [it modernization](<https://devfeed.tech/topics/it-modernization.md>), [event-driven data processing](<https://devfeed.tech/topics/event-driven-data-processing.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Jamstack](<https://devfeed.tech/topics/jamstack.md>)

Tags: [ach](<https://devfeed.tech/tags/ach.md>), [ach-fraud-monitoring](<https://devfeed.tech/tags/ach-fraud-monitoring.md>), [ach-modernization](<https://devfeed.tech/tags/ach-modernization.md>), [ach-payment-infrastructure](<https://devfeed.tech/tags/ach-payment-infrastructure.md>), [ach-payment-processing](<https://devfeed.tech/tags/ach-payment-processing.md>), [apis](<https://devfeed.tech/tags/apis.md>), [batch](<https://devfeed.tech/tags/batch.md>), [data](<https://devfeed.tech/tags/data.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [fraud](<https://devfeed.tech/tags/fraud.md>), [good-company](<https://devfeed.tech/tags/good-company.md>), [incremental](<https://devfeed.tech/tags/incremental.md>), [iso-20022](<https://devfeed.tech/tags/iso-20022.md>), [iso-20022-payments](<https://devfeed.tech/tags/iso-20022-payments.md>), [modernizing](<https://devfeed.tech/tags/modernizing.md>), [payment-modernization](<https://devfeed.tech/tags/payment-modernization.md>), [processing](<https://devfeed.tech/tags/processing.md>), [real-time-payment-architecture](<https://devfeed.tech/tags/real-time-payment-architecture.md>)

### AI overview

ACH volume and value are growing, while legacy batch processing, fixed-width files and disconnected risk systems make it harder for banks to meet intraday speed, data and fraud-monitoring demands. Arun Kumar recommends translating NACHA files into an ISO 20022-aligned internal payment model; processing entries as events and assembling files at each window; embedding account validation, behavioral analytics and OFAC screening before release; and offering corporate APIs for initiation, status and returns. He advises incremental modernization: wrap the legacy engine and replace returns, exceptions and reporting components as new services prove themselves, rather than attempting a big-bang core replacement. The excerpt frames these steps as a path to a unified, real-time-ready payments platform; it does not provide implementation details beyond the recommendations.

### Source excerpt

Learn how banks can modernize ACH with event-driven processing, ISO 20022-aligned data, APIs, integrated fraud controls, and incremental upgrades.

## Recognizing Events Is Not Enough: Why Streaming VLMs Struggle in Compositional Instruction Triggering

DevFeed: [Recognizing Events Is Not Enough: Why Streaming VLMs Struggle in Compositional Instruction Triggering](<https://devfeed.tech/articles/recognizing-events-is-not-enough-why-streaming-vlms-struggle-in-compositional-instruction-triggering-137337.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18743>)

Author: Yehua Huang, Penglei Sun, Xiang Li, Jinhao Zhou, Jiawen Wen, Qingqing Xiao, Zihan Yang, Wenqi Wang, Haolan Chen, Yu Xu, Xiaowen Chu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [after](<https://devfeed.tech/tags/after.md>), [agent](<https://devfeed.tech/tags/agent.md>), [atomic](<https://devfeed.tech/tags/atomic.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [device](<https://devfeed.tech/tags/device.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [event](<https://devfeed.tech/tags/event.md>), [events](<https://devfeed.tech/tags/events.md>), [example](<https://devfeed.tech/tags/example.md>), [execution](<https://devfeed.tech/tags/execution.md>), [existing](<https://devfeed.tech/tags/existing.md>), [fine](<https://devfeed.tech/tags/fine.md>), [flash](<https://devfeed.tech/tags/flash.md>), [frame](<https://devfeed.tech/tags/frame.md>), [gap](<https://devfeed.tech/tags/gap.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [rtx-5060](<https://devfeed.tech/tags/rtx-5060.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [vlms](<https://devfeed.tech/tags/vlms.md>)

### AI overview

Recognizing separate events is not enough for streaming video assistants: they must track when temporal conditions are met to notify at the right moment or remain silent. The paper introduces CoTriggerBench, which tests this using temporally constrained instructions, executable event plans, and replay of annotated source events. Its proposed real-time VLM agent, TEMPI, breaks instructions into event queries and temporal constraints, then uses Scenario Memory to track progress across observations beyond the detector's visual window; relevant events update that state even when no notification is due. On CoTriggerBench, TEMPI reports 44.4% instruction-level success, 15.6 percentage points above Qwen3.8-Omni-Flash-Realtime and 18.1 above MOSS-VL-Realtime. The authors also report a 4.90x speedup for an optimized detector on an 8 GB RTX 5060, using overlapping visual encoding and LLM prefix prefill with incoming frames.

### Source excerpt

arXiv:2611.18743v1 Announce Type: cross Abstract: Recognizing individual events does not guarantee correct execution of compositional instructions in streaming video. For example, given the request "Notify me when the pot boils after someone turns on the stove and leaves the kitchen", existing models may recognize the individual events yet notify too early or miss the intended moment, failing to track how their temporal conditions are satisfied over time. To evaluate this gap, we introduce CoTriggerBench, which composes annotated video events into temporally constrained instructions and verifies notification timing and silence through executable event plans and source-event replay. To address this challenge, we propose a real-time VLM agent, TEMPI. TEMPI decomposes instructions into atomic event queries and temporal constraints, and uses Scenario Memory to continuously maintain task progress as event evidence arrives, allowing earlier observations to support subsequent decisions beyond the detector's visual window. Even when no notification is required, relevant events continue to update the scenario state. On CoTriggerBench, TEMPI achieves 44.4% instruction-level success, outperforming the strongest closed-source and fine-tuned open-source VLM baselines by 15.6 (Qwen3.8-Omni-Flash-Realtime) and 18.1 (MOSS-VL-Realtime) percentage points, respectively. To support efficient on-device streaming inference, we further optimize the streaming inference pipeline by overlapping partial visual encoding and LLM prefix prefill with frame arrival. On an 8 GB RTX 5060, our optimized detector implementation achieves a 4.90$\times$ speedup.

## Build a real-time next-best-offer recommender using Databricks Feature Views

DevFeed: [Build a real-time next-best-offer recommender using Databricks Feature Views](<https://devfeed.tech/articles/build-a-real-time-next-best-offer-recommender-using-databricks-feature-views-136415.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/build-real-time-next-best-offer-recommender-using-databricks-feature-views>)

Author: Tian Tan; Sixuan He; Nick Joung; Julia Powell

Published: 2026-11-16T17:30:00Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [real-time](<https://devfeed.tech/topics/real-time.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [after](<https://devfeed.tech/tags/after.md>), [aggregate](<https://devfeed.tech/tags/aggregate.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [best](<https://devfeed.tech/tags/best.md>), [build](<https://devfeed.tech/tags/build.md>), [customers](<https://devfeed.tech/tags/customers.md>), [data-science-and-ml](<https://devfeed.tech/tags/data-science-and-ml.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [feature](<https://devfeed.tech/tags/feature.md>), [financial-services](<https://devfeed.tech/tags/financial-services.md>), [next](<https://devfeed.tech/tags/next.md>), [offer](<https://devfeed.tech/tags/offer.md>), [personalization](<https://devfeed.tech/tags/personalization.md>), [platform](<https://devfeed.tech/tags/platform.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [time](<https://devfeed.tech/tags/time.md>), [using](<https://devfeed.tech/tags/using.md>), [want](<https://devfeed.tech/tags/want.md>)

### AI overview

The accelerator shows how Databricks Feature Views can keep training and online serving feature definitions aligned for a retail banking next-best-offer ranker. Its `create_training_set` point-in-time joins reconstruct historical values without future leakage; for inference, Model Serving retrieves customer features from the online store. The notebooks start with batch features and add Kafka clickstream aggregates, avoiding separate hand-written streaming feature logic. In the stated benchmark setup, the ranking endpoint handled about 56 requests per second and stayed within a 50 ms p95 service budget; adding streaming features increased latency by 3-4 ms. Events became model-visible in about 0.1 seconds. Results used a small, untuned configuration, and scale-to-zero can cause cold starts of tens of seconds. The data is synthetic, so the accelerator does not establish revenue or conversion gains, and teams must add eligibility, consent, suitability, and suppression controls before production. Part 2 requires a Unity Catalog Kafka connection and Amazon MSK connectivity.

### Source excerpt

Real-time personalization depends on understanding what customers want in the moment...

## Rebuilding Guardian: How Pinterest Upgraded Its Real-Time Abuse Platform to Modern Infrastructure...

DevFeed: [Rebuilding Guardian: How Pinterest Upgraded Its Real-Time Abuse Platform to Modern Infrastructure...](<https://devfeed.tech/articles/rebuilding-guardian-how-pinterest-upgraded-its-real-time-abuse-platform-to-modern-infrastructure-136560.md>)

Original publisher: [Read original article](<https://medium.com/pinterest-engineering/rebuilding-guardian-how-pinterest-upgraded-its-real-time-abuse-platform-to-modern-infrastructure-56e45ee8f471?source=rss----4c5a5f6279b6---4>)

Author: Pinterest Engineering

Published: 2026-11-16T16:01:03Z

Content type: article

Language: en

Sources: [Pinterest](<https://devfeed.tech/sources/pinterest-engineering-blog-medium.md>)

Topics: [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>)

Tags: [account](<https://devfeed.tech/tags/account.md>), [account-takeover](<https://devfeed.tech/tags/account-takeover.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [blog](<https://devfeed.tech/tags/blog.md>), [cases](<https://devfeed.tech/tags/cases.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [content](<https://devfeed.tech/tags/content.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [data](<https://devfeed.tech/tags/data.md>), [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>), [detection](<https://devfeed.tech/tags/detection.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [flink](<https://devfeed.tech/tags/flink.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [modern](<https://devfeed.tech/tags/modern.md>), [pinterest](<https://devfeed.tech/tags/pinterest.md>), [platform](<https://devfeed.tech/tags/platform.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [starrocks](<https://devfeed.tech/tags/starrocks.md>), [trust-and-safety](<https://devfeed.tech/tags/trust-and-safety.md>)

### AI overview

Guardian helps Pinterest teams analyze real-time event data and turn identified patterns into signals, policies, and downstream actions for use cases such as spam and fraud response, account takeover detection, risk monitoring, and unsafe content enforcement. As Pinterest scaled, Guardian V1's monolithic design made it increasingly difficult to optimize interactive queries, continuous event-to-action processing, and backfills. Guardian V2 was rebuilt on modern data infrastructure to address scale, reliability, and compliance needs while preserving the fast workflow from investigation to operational response. The excerpt does not provide further architectural details.

### Source excerpt

Rebuilding Guardian: How Pinterest Upgraded Its Real-Time Abuse Platform to Modern Infrastructure at Scale Authors: Sharon Xie, Sr. Staff Software Engineer | Zhenxiao Luo, Sr Staff Software Engineer |Yi Pan, Principal Software Engineer | Vikram Deshpande, Sr. Engineering Manager Guardian helps Pinterest teams turn real-time event data into meaningful insights and operational outcomes. It enables subject matter experts to analyze real-time event streams, identify evolving patterns, and operationalize those learnings through signals, policies, and downstream workflows. Over the years, Guardian has supported a wide range of use cases, including spam and fraud response, account takeover detection, risk monitoring, and unsafe content enforcement. As Pinterest scaled, however, the original Guardian platform began to reach its limits. To meet growing demands around scale, reliability, and compliance, we rebuilt the system as Guardian V2 on top of modern data infrastructure. In this blog series, we'll share our journey in building the next generation of Guardian, or Guardian V2. In this first part, we'll discuss the challenges we faced, the architecture shift, and the foundational decisions that improved the platform's reliability, scalability and compliance alignment. Background & Motivation Guardian's biggest strength was the fast loop it enabled from investigation to operational response. In the asynchronous pipeline, teams could analyze enriched event data, identify suspicious patterns, convert those insights into actions, and trigger downstream actions quickly. Preserving that workflow was a core requirement for Guardian V2. As Pinterest scaled, however, the previous generation of Guardian presented several challenges: Scalability, scalability became a growing issue. Guardian V1 used a monolithic architecture to support interactive querying, continuous event to action processing, and backfill workflows, which made it increasingly difficult to optimize each path indepen

## From alerts to answers: The new Dynatrace Event Connector for ServiceNow ITOM

DevFeed: [From alerts to answers: The new Dynatrace Event Connector for ServiceNow ITOM](<https://devfeed.tech/articles/from-alerts-to-answers-the-new-dynatrace-event-connector-for-servicenow-itom-130714.md>)

Original publisher: [Read original article](<https://www.dynatrace.com/news/blog/from-alerts-to-answers-the-new-dynatrace-event-connector-for-servicenow-itom/>)

Author: Sharon Sharlin

Published: 2026-11-12T18:29:45Z

Content type: article

Language: en

Sources: [Dynatrace news](<https://devfeed.tech/sources/dynatrace-news.md>)

Topics: [dynatrace](<https://devfeed.tech/topics/dynatrace.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [payload](<https://devfeed.tech/topics/payload.md>), [Error Handling](<https://devfeed.tech/topics/error-handling.md>)

Tags: [alert-fatigue](<https://devfeed.tech/tags/alert-fatigue.md>), [alert-management](<https://devfeed.tech/tags/alert-management.md>), [alerts](<https://devfeed.tech/tags/alerts.md>), [automation](<https://devfeed.tech/tags/automation.md>), [central-teams](<https://devfeed.tech/tags/central-teams.md>), [ci](<https://devfeed.tech/tags/ci.md>), [cmdb](<https://devfeed.tech/tags/cmdb.md>), [context](<https://devfeed.tech/tags/context.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [delay](<https://devfeed.tech/tags/delay.md>), [dependency-graph](<https://devfeed.tech/tags/dependency-graph.md>), [dynatrace](<https://devfeed.tech/tags/dynatrace.md>), [event](<https://devfeed.tech/tags/event.md>), [events](<https://devfeed.tech/tags/events.md>), [incident](<https://devfeed.tech/tags/incident.md>), [incident-response](<https://devfeed.tech/tags/incident-response.md>), [it-operations](<https://devfeed.tech/tags/it-operations.md>), [problems](<https://devfeed.tech/tags/problems.md>), [product-news](<https://devfeed.tech/tags/product-news.md>), [root](<https://devfeed.tech/tags/root.md>), [servicenow](<https://devfeed.tech/tags/servicenow.md>), [shifting-left](<https://devfeed.tech/tags/shifting-left.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

Dynatrace's updated ServiceNow ITOM Event Connector uses a ready-to-use Workflows action to send detected problems, root-cause analysis, affected entities, Smartscape topology context, evidence, tags, severity, and impact to ServiceNow Event Management. Workflows map Dynatrace data to ServiceNow event fields; ServiceNow's existing policies can correlate events, associate configuration items, promote alerts, and escalate incidents. Configurable delays of 5 minutes to 7 days, retry handling, and support for up to 10,000 events per hour are described as controls for production event processing. The Service Graph Connector can help bind events to CMDB records. The post also describes a migration path for customers using the classic integration.

### Source excerpt

The new Dynatrace ITOM Event Connector sends problems detected by Dynatrace to ServiceNow Event Management. It helps teams correlate events, identify root causes, and accelerate incident response, while giving them an easy and flexible way to connect Dynatrace with ServiceNow Event Management. The post From alerts to answers: The new Dynatrace Event Connector for ServiceNow ITOM appeared first on Dynatrace news.

## Using Neo4j Change Data Capture to Orchestrate Agents

DevFeed: [Using Neo4j Change Data Capture to Orchestrate Agents](<https://devfeed.tech/articles/using-neo4j-change-data-capture-to-orchestrate-agents-130390.md>)

Original publisher: [Read original article](<https://neo4j.com/blog/developer/change-data-capture-to-orchestrate-agents/>)

Author: julian busch

Published: 2026-11-12T09:47:29Z

Content type: article

Language: en

Sources: [Neo4j](<https://devfeed.tech/sources/graph-database-technology-neo4j-blog.md>)

Topics: [agent orchestration](<https://devfeed.tech/topics/agent-orchestration.md>), [Neo4j](<https://devfeed.tech/topics/neo4j.md>), [graph-database](<https://devfeed.tech/topics/graph-database.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [agent to agent communication](<https://devfeed.tech/topics/agent-to-agent-communication.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [agent-orchestration](<https://devfeed.tech/tags/agent-orchestration.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agentic-workflow](<https://devfeed.tech/tags/agentic-workflow.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent-development](<https://devfeed.tech/tags/ai-agent-development.md>), [ai-and-context](<https://devfeed.tech/tags/ai-and-context.md>), [ai-and-machine-learning](<https://devfeed.tech/tags/ai-and-machine-learning.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cloud-orchestration](<https://devfeed.tech/tags/cloud-orchestration.md>), [cypher](<https://devfeed.tech/tags/cypher.md>), [data-orchestration](<https://devfeed.tech/tags/data-orchestration.md>), [developer](<https://devfeed.tech/tags/developer.md>), [knowledge-graphs](<https://devfeed.tech/tags/knowledge-graphs.md>), [multi-agent-orchestration](<https://devfeed.tech/tags/multi-agent-orchestration.md>), [neo4j](<https://devfeed.tech/tags/neo4j.md>)

### AI overview

Neo4j CDC can trigger agent handoffs directly from committed graph changes: a poller reads ordered change records with `db.cdc.query`, filters for transitions such as a `Task` moving from `PENDING` to `ACTIVE`, then dispatches an agent whose write-back can trigger the next step. This keeps task state, events, governance, and audit data queryable together in Cypher and avoids reconciling a separate broker. The excerpt describes graph-based dead-letter records, cursor replay within the transaction log's readable window, single-poller leases, and idempotent dispatch. Trade-offs include a roughly 750 ms polling floor per hop and possible loss if downtime exceeds log retention; Kafka is suggested for durable retention, while Temporal can handle durable timers or saga coordination. CDC requires AuraDB Business Critical or a self-managed Enterprise or Infinigraph edition.

### Source excerpt

You've built a multi-agent orchestration pipeline. One agent finishes its work and writes a result. The next agent should pick that result up and run. Something has to notice the first agent finished and wake the second one. The usual... Read more ->

## SepsisFlow: an event-stream foundation model for continuous risk warning and trajectory modelling in ICU sepsis

DevFeed: [SepsisFlow: an event-stream foundation model for continuous risk warning and trajectory modelling in ICU sepsis](<https://devfeed.tech/articles/sepsisflow-an-event-stream-foundation-model-for-continuous-risk-warning-and-trajectory-modelling-in-icu-sepsis-128746.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13661>)

Author: Yawen Zhang, Wanzhe Xu, Zhenhua Zhang, Jinyu Wu, Yizhou Zhuang, Yitao Yang, Kenta Nakai

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>)

Tags: [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attention](<https://devfeed.tech/tags/attention.md>), [combines](<https://devfeed.tech/tags/combines.md>), [context](<https://devfeed.tech/tags/context.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [course](<https://devfeed.tech/tags/course.md>), [courses](<https://devfeed.tech/tags/courses.md>), [critical](<https://devfeed.tech/tags/critical.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [death](<https://devfeed.tech/tags/death.md>), [event](<https://devfeed.tech/tags/event.md>), [features](<https://devfeed.tech/tags/features.md>), [foundation-model](<https://devfeed.tech/tags/foundation-model.md>), [gradient-boosted-trees](<https://devfeed.tech/tags/gradient-boosted-trees.md>), [it](<https://devfeed.tech/tags/it.md>), [mamba](<https://devfeed.tech/tags/mamba.md>), [modelling](<https://devfeed.tech/tags/modelling.md>), [models](<https://devfeed.tech/tags/models.md>), [risk](<https://devfeed.tech/tags/risk.md>), [stream](<https://devfeed.tech/tags/stream.md>), [warning](<https://devfeed.tech/tags/warning.md>)

### AI overview

SepsisFlow models sepsis on an hourly timeline spanning warning, onset, and early course. Before onset, it combines a Mamba model pretrained on 2.57 million EHRs with about 1,700 hourly features in gradient-boosted trees to estimate 6-, 12-, and 24-hour risk. At onset, a gated network fuses Mamba and tabular encoders to define patient contexts; after onset, an attention-based bidirectional recurrent network encodes 72-hour organ-support and physiological trajectories. In MIMIC-IV, it detected more cases than NEWS2, qSOFA, and SIRS at equal alarms, with hourly updating adding about 8 cases per 100, and outperformed 16 retrained algorithms. Risk ranking was maintained in eICU. Reported 28-day mortality differed between onset contexts (2% in cardiovascular ICU versus 15%-17% in others); within each context, persistent support was associated with fivefold higher mortality than de-escalation. The evidence provided is an abstract, so it does not detail methods or limitations beyond these claims.

### Source excerpt

arXiv:2611.13661v1 Announce Type: new Abstract: Sepsis is a leading cause of ICU death with heterogeneous onset and course, yet most warning models predict only whether sepsis will occur, leaving clinicians uncertain of its onset context and early trajectory once an alarm sounds. To answer these questions on one hourly clock from ICU admission, we built SepsisFlow: before onset, it combines a Mamba model pretrained on 2.57 million EHRs with about 1,700 hourly features in gradient-boosted trees to predict 6/12/24-h risk; at onset, it fuses Mamba and tabular encoders via a gated network into continuous patient states defining onset contexts; after onset, it encodes 72-h organ-support and physiological trajectories via an attention bidirectional recurrent network into early courses. For pre-onset warning in MIMIC-IV, SepsisFlow detected more cases than NEWS2, qSOFA, and SIRS at equal alarms, gaining about 8 cases per 100 from hourly updating, while outperforming 16 retrained algorithms and maintaining risk ranking in eICU. At onset, its contexts separated 2% cardiovascular-ICU from 15%-17% other 28-day mortality, and within each context after onset, persistent support carried fivefold higher mortality than de-escalation. SepsisFlow establishes a foundation-model paradigm for continuous risk and trajectory stratification in critical care.

## Debezium 3.6.4.Final Released

DevFeed: [Debezium 3.6.4.Final Released](<https://devfeed.tech/articles/debezium-3-6-4-final-released-129189.md>)

Original publisher: [Read original article](<https://debezium.io/blog/2026/11/10/debezium-3-6-4-final-released/>)

Author: Chris Cranford

Published: 2026-11-10T00:00:00Z

Content type: release

Language: en

Sources: [Debezium](<https://devfeed.tech/sources/debezium-io.md>)

Topics: [Debezium Connector](<https://devfeed.tech/topics/debezium-connector.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>)

Tags: [capture](<https://devfeed.tech/tags/capture.md>), [change](<https://devfeed.tech/tags/change.md>), [cve](<https://devfeed.tech/tags/cve.md>), [debezium](<https://devfeed.tech/tags/debezium.md>), [drivers](<https://devfeed.tech/tags/drivers.md>), [events](<https://devfeed.tech/tags/events.md>), [faster](<https://devfeed.tech/tags/faster.md>), [latest](<https://devfeed.tech/tags/latest.md>), [maintenance](<https://devfeed.tech/tags/maintenance.md>), [mongodb](<https://devfeed.tech/tags/mongodb.md>), [mysql](<https://devfeed.tech/tags/mysql.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [published](<https://devfeed.tech/tags/published.md>), [query](<https://devfeed.tech/tags/query.md>), [release](<https://devfeed.tech/tags/release.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [security](<https://devfeed.tech/tags/security.md>), [security-updates](<https://devfeed.tech/tags/security-updates.md>), [series](<https://devfeed.tech/tags/series.md>), [tables](<https://devfeed.tech/tags/tables.md>)

### AI overview

Debezium 3.6.4.Final is a maintenance release with 35 resolved issues, led by Oracle reliability fixes for missed change events on RAC and a more efficient LogMiner query. The query reduces database CPU overhead and streams results faster, with greater benefit for deployments capturing hundreds of tables. The MongoDB Java driver is upgraded to 5.14.0 with no configuration changes required. MySQL Connector/J moves to 26.10.0 to address CVE-2025-30706; most deployments need no changes, but its documented support range starts at MySQL Server 8.4, and custom driver keystores must be local files. The release also includes fixes across Oracle, SQL Server, MySQL, and shared components.

### Source excerpt

We're pleased to announce Debezium 3.6.4.Final, the latest maintenance release of the 3.6 series, focused on stability and reliability. This release is led by a round of fixes for the Oracle connector, resolving several cases where change events could be missed, most notably on Oracle RAC. Oracle deployments that capture many tables also benefit from a faster LogMiner query. Beyond Oracle, the release brings security updates through newer MySQL and MongoDB drivers, including a MySQL driver that addresses a published CVE. Read on for the full details! Read more...

## Real-time state has a cost your memory bill won't show

DevFeed: [Real-time state has a cost your memory bill won't show](<https://devfeed.tech/articles/real-time-state-has-a-cost-your-memory-bill-won-t-show-123756.md>)

Original publisher: [Read original article](<https://thenewstack.io/duplicated-streaming-state-costs/>)

Author: Granville Barnett

Published: 2026-11-09T14:29:57Z

Content type: article

Language: en

Sources: [The New Stack](<https://devfeed.tech/sources/the-new-stack.md>)

Topics: [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [cloud cost management](<https://devfeed.tech/topics/cloud-cost-management.md>)

Tags: [17](<https://devfeed.tech/tags/17.md>), [2](<https://devfeed.tech/tags/2.md>), [applications](<https://devfeed.tech/tags/applications.md>), [backend-development](<https://devfeed.tech/tags/backend-development.md>), [bill](<https://devfeed.tech/tags/bill.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data-streaming](<https://devfeed.tech/tags/data-streaming.md>), [databases](<https://devfeed.tech/tags/databases.md>), [events](<https://devfeed.tech/tags/events.md>), [first](<https://devfeed.tech/tags/first.md>), [flink](<https://devfeed.tech/tags/flink.md>), [handle](<https://devfeed.tech/tags/handle.md>), [hazelcast](<https://devfeed.tech/tags/hazelcast.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [memory](<https://devfeed.tech/tags/memory.md>), [post](<https://devfeed.tech/tags/post.md>), [process](<https://devfeed.tech/tags/process.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [roles](<https://devfeed.tech/tags/roles.md>), [setup](<https://devfeed.tech/tags/setup.md>), [show](<https://devfeed.tech/tags/show.md>), [sponsor-hazelcast](<https://devfeed.tech/tags/sponsor-hazelcast.md>), [sponsored](<https://devfeed.tech/tags/sponsored.md>), [sponsored-post](<https://devfeed.tech/tags/sponsored-post.md>), [state](<https://devfeed.tech/tags/state.md>), [time](<https://devfeed.tech/tags/time.md>), [tools](<https://devfeed.tech/tags/tools.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Separating stream processing from serving can add CPU, network, synchronization, and infrastructure costs when systems maintain overlapping state. In the example, Flink keeps keyed state for a five-minute payment velocity count, while Redis serves the count to applications; updates must be serialized and published, creating export lag. Replica lag is a separate source of stale reads, and neither issue alone explains a check-then-act race that could duplicate a refund. Hazelcast offers entry processors to update map entries where they are stored, or Jet for windowed calculations that can publish results to a distributed map. These approaches can reduce cross-system movement, but backups, member traffic, serialization, late or duplicate events, and consistency requirements remain relevant. Separate platforms may still suit workloads needing independent scaling, isolation, or specialized processing. The excerpt recommends profiling CPU and network use, duplicated state, tail and end-to-end latency, and testing recovery and correctness under failures using comparable workloads and durability targets. Its cited cost and latency studies are not direct Hazelcast-versus-Flink/Redis savings

### Source excerpt

Real-time applications often combine tools with distinct roles. A typical setup might use Kafka to handle events, Flink to process The post Real-time state has a cost your memory bill won't show appeared first on The New Stack.

## Webhooks to Kafka: How to Receive Events and Publish Them to Topics

DevFeed: [Webhooks to Kafka: How to Receive Events and Publish Them to Topics](<https://devfeed.tech/articles/webhooks-to-kafka-how-to-receive-events-and-publish-them-to-topics-121710.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/webhooks-to-kafka>)

Author: Seán Whelan

Published: 2026-11-06T12:27:07Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [compare](<https://devfeed.tech/tags/compare.md>), [events](<https://devfeed.tech/tags/events.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [http](<https://devfeed.tech/tags/http.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [managed](<https://devfeed.tech/tags/managed.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [retries](<https://devfeed.tech/tags/retries.md>), [webhooks](<https://devfeed.tech/tags/webhooks.md>)

### AI overview

Webhook providers cannot post directly to Kafka, so an HTTPS receiver must verify each request and publish it. The excerpt compares a custom service or HTTP gateway, API Gateway with Lambda, Kafka Connect, and a managed pipeline; the choice depends on operational control, AWS or Connect infrastructure, and who hosts the endpoint. For a direct receiver, verify signatures over raw bytes, produce with the delivery ID as key, and return success only after Kafka acknowledges; provider retries make delivery at least once, so redeliveries can become duplicate messages. Providers have different deadlines and retry behavior--Shopify allows five seconds, while GitHub does not automatically redeliver. Estuary instead acknowledges after committing requests to a durable collection, then materializes them to Kafka independently, allowing capture to continue during a broker outage. Its excerpted limits include a 64 MiB request cap, message truncation thresholds, and signature-header formats that do not support GitHub's or Stripe's built-in formats through the custom setting. The text also covers Kafka size and authentication requirements, recovery, deduplication, and alternate routes.

### Source excerpt

Send webhooks to Kafka with a webhook source connector, an HTTP gateway, AWS Lambda or a managed pipeline. Compare signatures, retries and duplicates.

## Kafka to ClickHouse: Streaming Ingestion, Duplicates & Tombstones Guide

DevFeed: [Kafka to ClickHouse: Streaming Ingestion, Duplicates & Tombstones Guide](<https://devfeed.tech/articles/kafka-to-clickhouse-streaming-ingestion-duplicates-tombstones-guide-121703.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/kafka-to-clickhouse>)

Author: Seán Whelan

Published: 2026-11-06T12:26:41Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [self-hosted databases vs managed services](<https://devfeed.tech/topics/self-hosted-databases-vs-managed-services.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>)

Tags: [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [compare](<https://devfeed.tech/tags/compare.md>), [connect](<https://devfeed.tech/tags/connect.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [engine](<https://devfeed.tech/tags/engine.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [guide](<https://devfeed.tech/tags/guide.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [stream](<https://devfeed.tech/tags/stream.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [table](<https://devfeed.tech/tags/table.md>)

### AI overview

Kafka ingestion into ClickHouse is generally at least once, so replayed messages can create duplicates; reliable designs need batching and tables suited to event or keyed data. The article compares the Kafka table engine, ClickPipes, the Kafka Connect sink, and managed pipelines: the engine runs consumers on ClickHouse servers, ClickPipes is recommended for ClickHouse Cloud, and the sink fits teams already running Kafka Connect. For event streams, store partition and offset in a ReplacingMergeTree sorting key; keyed topics with tombstones need ReplacingMergeTree keyed by message key and queries with FINAL. Tombstone support differs: ClickPipes can delete rows in exactly-once mode or mark them deleted, while the Connect sink lists deletes as unsupported and the table engine's documentation does not describe null-value messages. Retention limits recovery, and compaction can remove tombstones after the configured retention period. Large, infrequent inserts reduce part pressure; monitor lag and active parts. The article also describes Estuary capture and materialization, which keeps Kafka progress in its own checkpoints and writes to ClickHouse on a configurable schedule.

### Source excerpt

Stream Kafka to ClickHouse with the Kafka table engine, ClickPipes or Kafka Connect. Compare 4 methods, delivery guarantees and duplicate handling.

## Change Data Capture in SQLite, Without the Triggers

DevFeed: [Change Data Capture in SQLite, Without the Triggers](<https://devfeed.tech/articles/change-data-capture-in-sqlite-without-the-triggers-121133.md>)

Original publisher: [Read original article](<https://turso.tech/blog/change-data-capture-in-sqlite-without-the-triggers>)

Author: Jeff Olson

Published: 2026-11-05T00:00:00Z

Content type: article

Language: en

Sources: [Turso](<https://devfeed.tech/sources/turso-blog.md>)

Topics: [Turso](<https://devfeed.tech/topics/turso.md>), [SQLite](<https://devfeed.tech/topics/sqlite.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [event-driven data processing](<https://devfeed.tech/topics/event-driven-data-processing.md>), [transaction-log](<https://devfeed.tech/topics/transaction-log.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Debezium Connector](<https://devfeed.tech/topics/debezium-connector.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [built-in](<https://devfeed.tech/tags/built-in.md>), [capture](<https://devfeed.tech/tags/capture.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [change](<https://devfeed.tech/tags/change.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [delete](<https://devfeed.tech/tags/delete.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [here](<https://devfeed.tech/tags/here.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [insert](<https://devfeed.tech/tags/insert.md>), [records](<https://devfeed.tech/tags/records.md>), [software-architecture](<https://devfeed.tech/tags/software-architecture.md>), [sqlite](<https://devfeed.tech/tags/sqlite.md>), [triggers](<https://devfeed.tech/tags/triggers.md>), [turso](<https://devfeed.tech/tags/turso.md>), [update](<https://devfeed.tech/tags/update.md>), [works](<https://devfeed.tech/tags/works.md>), [write](<https://devfeed.tech/tags/write.md>)

### AI overview

Turso Database adds built-in SQLite CDC that records committed inserts, updates, and deletes in `turso_cdc` without per-table triggers. Consumers can poll the ordered stream, group records by transaction, and resume from a saved watermark; they should make processing idempotent because crashes can replay already-applied changes. Modes retain primary keys, before or after images, or full details, and schema changes are recorded. The feature is actively developed: pin versions, and snapshot column lists before migrations because decoding old records with today's columns can misalign them. CDC captures row facts, not domain meaning or user identity; those require application-side context. The excerpt also cautions that retention cleanup on synced databases may remove changes Sync has not pushed.

### Source excerpt

Built-in CDC in Turso Database records every insert, update, and delete for you. Here is how it works, where it is rough, and how to write a consumer that survives restarts.

## Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19

DevFeed: [Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19](<https://devfeed.tech/articles/introducing-apache-spark-4-3-now-available-in-databricks-runtime-19-117923.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/introducing-apache-spark-43-now-available-databricks-runtime-19>)

Author: Jungtaek Lim; Wenchen Fan; Andreas Neumann; Serge Rielau; Hyukjin Kwon; Jerry Peng; DB Tsai; Daniel Tenedorio; Xiao Li; Reynold Xin

Published: 2026-11-05T00:00:00Z

Content type: release

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Stateful](<https://devfeed.tech/topics/stateful.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>), [apache-arrow](<https://devfeed.tech/topics/apache-arrow.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [spark](<https://devfeed.tech/tags/spark.md>), [stateful](<https://devfeed.tech/tags/stateful.md>)

### AI overview

Apache Spark 4.3 extends Structured Streaming's Real-Time Mode with stateful aggregation, deduplication and custom processing, reporting P99 end-to-end latency below 150 ms across three tested benchmarks; those measurements are not a guarantee for all workloads. It enables continuous data flow through concurrent stage scheduling and pipelined shuffles, while retaining Spark's state store and checkpointing. The release also adds Auto CDC SCD Type 2 history through Python, SQL and Spark Connect, requiring a catalog connector that supports Data Source V2 row-level operations (built-in Parquet does not). Other additions include opt-in shuffle-free execution for eligible small queries, Spark Connect session and operation visibility, JSON and VARIANT processing, and ASOF JOIN and nanosecond timestamp previews that may change. Spark 4.3 starts a quarterly feature-release cadence.

### Source excerpt

IntroductionApache Spark 4.3 brings stateful processing to Real-Time Mode, achieving...

## Sampling Ad Exchange Events at Scale: What We Kept, What We Dropped, and Why

DevFeed: [Sampling Ad Exchange Events at Scale: What We Kept, What We Dropped, and Why](<https://devfeed.tech/articles/sampling-ad-exchange-events-at-scale-what-we-kept-what-we-dropped-and-why-112261.md>)

Original publisher: [Read original article](<https://hackernoon.com/sampling-ad-exchange-events-at-scale-what-we-kept-what-we-dropped-and-why?source=rss>)

Author: Vikrant Bachhav

Published: 2026-11-02T06:00:04Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>)

Tags: [2019](<https://devfeed.tech/tags/2019.md>), [2022](<https://devfeed.tech/tags/2022.md>), [2023](<https://devfeed.tech/tags/2023.md>), [adtech](<https://devfeed.tech/tags/adtech.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [bi](<https://devfeed.tech/tags/bi.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [billing](<https://devfeed.tech/tags/billing.md>), [business](<https://devfeed.tech/tags/business.md>), [click](<https://devfeed.tech/tags/click.md>), [cloud-cost-optimization](<https://devfeed.tech/tags/cloud-cost-optimization.md>), [costs](<https://devfeed.tech/tags/costs.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-sampling-techniques](<https://devfeed.tech/tags/data-sampling-techniques.md>), [events](<https://devfeed.tech/tags/events.md>), [exchange](<https://devfeed.tech/tags/exchange.md>), [executor](<https://devfeed.tech/tags/executor.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [reporting](<https://devfeed.tech/tags/reporting.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [scale](<https://devfeed.tech/tags/scale.md>), [spark](<https://devfeed.tech/tags/spark.md>), [stream](<https://devfeed.tech/tags/stream.md>), [unity](<https://devfeed.tech/tags/unity.md>), [volume](<https://devfeed.tech/tags/volume.md>), [workers](<https://devfeed.tech/tags/workers.md>)

### AI overview

Unity reduced ad exchange stream volume from about 10 GB/s to 2 GB/s by sampling pre-filter events at 0.5% while retaining post-filter events at full fidelity. Sampling by auction ID kept related events joinable; counts from sampled data were scaled by the inverse sampling probability, while ratios remained comparable when both parts used the same sample. After reviewing reporting needs with report owners, the team moved reports to a separate sampled Kafka topic and tuned Spark executors. Worker count fell from 240 to 30 and CPU utilization rose 41%; DSP Report transform billing was about 41% lower for March 9-April 6, 2023 than the prior period. Two weeks of parallel comparisons preceded report cutover. Validation also uncovered longstanding join duplication that inflated BidWin and WinPrice metrics. The account notes that dimensions considered unsafe to sample were kept at full fidelity.

### Source excerpt

How we reduced Kafka stream volume by 80%, cut Spark workers from 240 to 30, and lowered reporting costs through sampling and executor tuning at Unity.

[Next page](<https://devfeed.tech/topics/event-streaming.md?cursor=WyIyMDI2LTExLTAyVDA2OjAwOjA0KzAwOjAwIiwgIjk1YTQ5OThiLTQwNTctNGI5MC1iM2YwLWE3MzllMmQ4MDgzNSJd>)