KIP-1150: Diskless Kafka Topics
5 comments
·April 16, 2025NortySpock
enether
Somewhat. The buying further validated the market, but at the time of purchase (Sep 2024) there already were 5 alternative proprietary solutions utilizing the same direct-to-S3 design shipped or announced (Confluent Freight, Bufstream, AutoMQ, Redpanda cloud topics, StreamNative Ursa). The cat was out of the bag that this is a great architecture
chtefi
Full disaggregation of compute and storage is the right direction. Let storage handle replication, it's getting good, global, low latency, cheaper (like with S3 Express). Kafka becomes a smart data ingester and router: it moves bytes, enforces ordering, does minimal buffering. That's it. Do one thing well.
You get a system simpler to operate, to scale, and more flexible; data could be consumed outside of Kafka itself (in a batch way typically), without duplicating the data, that's a big win.
wanshao
[dead]
null
I presume this is part of the knock-on effects of Confluent (managed Kafka) buying WarpStream (Kafka emulated on S3 object storage).
Also a shout-out to Bento, the fork of Benthos after RedPanda acquired it.