# How we built reliable log delivery to thousands of unpredictable endpoints

DevFeed: [How we built reliable log delivery to thousands of unpredictable endpoints](<https://devfeed.tech/articles/how-we-built-reliable-log-delivery-to-thousands-of-unpredictable-endpoints-83493.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/engineering/reliable-log-delivery/>)

Author: Gabriel Reid

Published: 2025-06-23T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog.md>)

Topics: [log management](<https://devfeed.tech/topics/log-management.md>), [Cloud and systems](<https://devfeed.tech/topics/cloud-and-systems.md>), [VPC Flow Logs](<https://devfeed.tech/topics/vpc-flow-logs.md>)

Tags: [batching](<https://devfeed.tech/tags/batching.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [data-loss](<https://devfeed.tech/tags/data-loss.md>), [latency](<https://devfeed.tech/tags/latency.md>), [log-management](<https://devfeed.tech/tags/log-management.md>)

## AI overview

Datadog describes a log forwarding system that groups records by destination in compressed slice files stored in cloud object storage, then uses planning and shipping services to deliver batches. Failed deliveries are retried with exponential backoff, while adaptive concurrency helps destinations recover without disrupting forwarding to healthy endpoints. The design isolates destination failures and prioritizes newer logs when processing backlogs.

## Source excerpt

Learn how we built Datadog's Log Forwarding system for low-latency, high-throughput delivery to thousands of unreliable third-party endpoints.