# Achieving relentless Kafka reliability at scale with the Streaming Platform

DevFeed: [Achieving relentless Kafka reliability at scale with the Streaming Platform](<https://devfeed.tech/articles/achieving-relentless-kafka-reliability-at-scale-with-the-streaming-platform-83510.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/engineering/streaming-platform-kafka-custom-abstractions/>)

Author: Guillaume Bort

Published: 2025-02-19T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [Resilience](<https://devfeed.tech/topics/resilience.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>)

Tags: [availability](<https://devfeed.tech/tags/availability.md>), [event-platform](<https://devfeed.tech/tags/event-platform.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [scale](<https://devfeed.tech/tags/scale.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

## AI overview

Datadog describes its Streaming Platform, which abstracts Kafka across clusters and environments to keep data pipelines reliable at large scale. It explains how Streams enable live failovers, the Assigner handles rapid rebalancing, lanes and an enhanced commit log reduce blocking, and libstreaming provides a custom Rust Kafka client with consistent performance and observability. The post also covers time-lag monitoring and ongoing work on storage resilience and commit-log design.

## Source excerpt

Learn how we built the Streaming Platform at Datadog to provide more resilience and flexibility to our Kafka infrastructure.