# Effective batching in streaming dataflow jobs

DevFeed: [Effective batching in streaming dataflow jobs](<https://devfeed.tech/articles/effective-batching-in-streaming-dataflow-jobs-15619.md>)

Original publisher: [Read original article](<https://developer.squareup.com/blog/effective-batching-in-streaming-dataflow-jobs>)

Author: Fasih Awan

Published: 2019-07-15T11:00:00Z

Content type: article

Language: en

Sources: [Square Corner Blog RSS Feed](<https://devfeed.tech/sources/square-corner-blog-rss-feed.md>)

Topics: [Streaming](<https://devfeed.tech/topics/streaming.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Time Series](<https://devfeed.tech/topics/time-series.md>), [Google](<https://devfeed.tech/topics/google.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [batching](<https://devfeed.tech/tags/batching.md>), [data](<https://devfeed.tech/tags/data.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [payload](<https://devfeed.tech/tags/payload.md>), [payments](<https://devfeed.tech/tags/payments.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

## AI overview

Square describes how it batches payment messages in a Google Dataflow job to produce near-real-time monitoring metrics. The pipeline consumes PubSub messages, applies five-second windows, aggregates payments by dimensions, and sends metrics to SignalFX while accounting for API restrictions.

## Source excerpt

How we make near real time monitoring possible