# Designing a Cloud-Native Event Log Beyond Kafka's Partition Model

DevFeed: [Designing a Cloud-Native Event Log Beyond Kafka's Partition Model](<https://devfeed.tech/articles/what-if-we-could-rebuild-kafka-from-scratch-18888.md>)

Original publisher: [Read original article](<https://www.morling.dev/blog/what-if-we-could-rebuild-kafka-from-scratch/>)

Published: 2025-04-24T14:25:00Z

Content type: opinion

Language: en

Sources: [Gunnar Morling](<https://devfeed.tech/sources/gunnar-morling.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [cloud-infrastructure](<https://devfeed.tech/topics/cloud-infrastructure.md>), [event driven](<https://devfeed.tech/topics/event-driven.md>), [systems](<https://devfeed.tech/topics/systems.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [cloud-native](<https://devfeed.tech/tags/cloud-native.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [event-sourcing](<https://devfeed.tech/tags/event-sourcing.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [partition](<https://devfeed.tech/tags/partition.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [s3](<https://devfeed.tech/tags/s3.md>)

## AI overview

The article proposes a personal wishlist for a durable cloud-native event log designed beyond Kafka's traditional architecture. It discusses separating storage and compute, using object storage such as S3, removing partitions, and providing key-centric streams with ordering and dynamically scalable consumers.

## Source excerpt

The last few days I spent some time digging into the recently announced KIP-1150 ("Diskless Kafka"), as well AutoMQ's Kafka fork, tightly integrating Apache Kafka and object storage, such as S3. Following the example set by WarpStream, these projects aim to substantially improve the experience of using Kafka in cloud environments, providing better elasticity, drastically reducing cost, and paving the way towards native lakehouse integration. This got me thinking, if we were to start all over and develop a durable cloud-native event log from scratch--Kafka.next if you will--which traits and characteristics would be desirable for this to have? Separating storage and compute and object store support would be table stakes, but what else should be there? Having used Kafka for many years for building event-driven applications as well as for running realtime ETL and change data capture pipelines, here's my personal wishlist: