# Smart Batching

DevFeed: [Smart Batching](<https://devfeed.tech/articles/smart-batching-13618.md>)

Original publisher: [Read original article](<https://mechanical-sympathy.blogspot.com/2011/10/smart-batching.html>)

Author: Martin Thompson (noreply@blogger.com)

Published: 2011-10-19T16:44:00Z

Content type: tutorial

Language: en

Sources: [Mechanical Sympathy](<https://devfeed.tech/sources/mechanical-sympathy.md>)

Topics: [Latency](<https://devfeed.tech/topics/latency.md>), [Low Latency](<https://devfeed.tech/topics/low-latency.md>), [Algorithm](<https://devfeed.tech/topics/algorithm.md>), [Code](<https://devfeed.tech/topics/code.md>), [IO](<https://devfeed.tech/topics/io.md>), [systems](<https://devfeed.tech/topics/systems.md>), [Data structures](<https://devfeed.tech/topics/data-structures.md>), [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [Network](<https://devfeed.tech/topics/network.md>), [Concurrent Programming](<https://devfeed.tech/topics/concurrent-programming.md>), [Filesystems](<https://devfeed.tech/topics/filesystems.md>), [Database](<https://devfeed.tech/topics/database.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [batching](<https://devfeed.tech/tags/batching.md>), [code](<https://devfeed.tech/tags/code.md>), [data](<https://devfeed.tech/tags/data.md>), [data-structures](<https://devfeed.tech/tags/data-structures.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [io](<https://devfeed.tech/tags/io.md>), [java](<https://devfeed.tech/tags/java.md>), [latency](<https://devfeed.tech/tags/latency.md>), [locks](<https://devfeed.tech/tags/locks.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [network](<https://devfeed.tech/tags/network.md>), [networking](<https://devfeed.tech/tags/networking.md>), [performance](<https://devfeed.tech/tags/performance.md>), [queue](<https://devfeed.tech/tags/queue.md>), [systems](<https://devfeed.tech/tags/systems.md>), [thread](<https://devfeed.tech/tags/thread.md>), [threads](<https://devfeed.tech/tags/threads.md>)

## AI overview

The article explains how correctly designed batching can improve throughput while reducing and stabilizing average latency. It discusses batching messages for network packets and storage writes, and presents a Java approach that sends immediately when data is available while grouping bursts up to a buffer limit. The approach can avoid lock contention, but an unbounded queue may require size tracking and back pressure.

## Source excerpt

How often have we all heard that "batching" will increase latency? As someone with a passion for low-latency systems this surprises me. In my experience when batching is done correctly, not only does it increase throughput, it can also reduce average latency and keep it consistent. Well then, how can batching magically reduce latency? It comes down to what algorithm and data structures are employed. In a distributed environment we are often having to batch up messages/events into network packets to achieve greater throughput. We also employ similar techniques in buffering writes to storage to reduce the number of IOPS. That storage could be a block device backed file-system or a relational database. Most IO devices can only handle a modest number of IO operations per second, so it is best to fill those operations efficiently. Many approaches to batching involve waiting for a timeout to occur and this will by its very nature increase latency. The batch can also get filled before the timeout occurs making the latency even more unpredictable. Figure 1. Figure 1. above depicts decoupling the access to an IO device, and therefore the contention for access to it, by introducing a queue like structure to stage the messages/events to be sent and a thread doing the batching for writing to the device. The Algorithm An approach to batching uses the following algorithm in Java pseudo code: public final class NetworkBatcher implements Runnable { private final NetworkFacade network; private final Queue<Message> queue; private final ByteBuffer buffer; public NetworkBatcher(final NetworkFacade network, final int maxPacketSize, final Queue<Message> queue) { this.network = network; buffer = ByteBuffer.allocate(maxPacketSize); this.queue = queue; } public void run() { while (!Thread.currentThread().isInterrupted()) { while (null == queue.peek()) { employWaitStrategy(); // block, spin, yield, etc. } Message msg; while (null != (msg = queue.poll())) { if (msg.size() > buffer.remaining()