🚨BREAKING: Direct-to-S3 topics are coming to Apache Kafka! KIP-1150: Diskless Topics was just published on the mailing list. After WarpStream pioneered this game-changer in 2023, everyone else -- Confluent, Bufstream, AutoMQ, RedPanda -- has been busy implementing their own version. Now it’s Kafka’s turn to enjoy 80%+ reduced infrastructure costs. The KIP proposes extending Kafka with so-called "Diskless Topics", which are: • leaderless - all brokers can accept writes for the same partition • directly persisted in object storage (pluggable interface) - no replicating between brokers If accepted, Kafka will become one of the only Kafka API-compatible systems that cater to both sides of the latency spectrum. It would support expensive, low-latency topics and cheap, high-latency topics. I think this feature is a no-brainer. The community must accept it one way or another. The benefits are simply too many to iterate: • no replication networking cost 💸 • no producer networking costs - producers can send writes within the same AZ because any broker can be a leader 💸 • easy to balance - since there is no local state and no concrete leader, clients can be easily and instantly moved to other brokers when hot spots form 👌 • fast scalability - add or remove brokers in seconds, as if Kafka is nginx 🤩 With it, Kafka becomes a very flexible system that can work efficiently both on-premise and in the cloud. I will be diving a lot more into the details of how it’s proposed to work, and the expected cost/ops savings. I think it can truly reduce costs by 10x (yes, ten times less). 💰 ...The architecture simply allows for it. 🤷‍♂️ Some vendors claim the same, but after their margins, it's not any cheaper than classic Kafka at all. With this feature being open source - you can be sure it actually will be. Diskless topics also pave the way for a lot of further innovation, as mentioned in the KIP: • Iceberg topics - with a pluggable first-class storage and partition leadership interface (the batch coordinator), adding a different log storage format can be done perhaps entirely via pluggable code 🧊 • parallel produce - increase throughput massively by processing a lot of multiple Produce requests concurrently. This is doable now because a broker won’t be bottlenecked by having to sequentially write the data in a single disk. 🔥 • multi-region active-active deployments via the underlying object store's cross-region replication, similar to how Bufstream does it 🌎 • using quorum S3 Express writes to achieve faster (double-digit) latency while still being stateless (similar to how WarpStream does it) 💥 • read-only replica clusters - once the data is all in S3 it becomes trivial to spin up a separate cluster to read from it (similar to how RedPanda does it) The team at Aiven is driving this. 🦀 The KIP seems well thought out! Apparently they’ve been building this and are deciding to open source it. I think it’s a brilliant strategic move for them. Keeping a fork would make it "yet another Kafka", but when you merge it to the open source - it becomes THE Kafka. Distribution is everything in business - once this (killer) feature is merged, it’ll go directly into the hands of the estimated 100,000+ companies using Apache Kafka. From the open source project's POV, there is a lot of wisdom in waiting for the free market to innovate, test and filter out viable ideas. It benefits from the second-mover advantage. In established projects like Kafka, the downside of getting it wrong can be greater than the upside of being first. 💡 Commercial vendors can therefore be seen as Kafka’s R&D department. They absorb the initial risks and costs of innovation, as well as perform distributed idea validation from their multiple different implementations. 🤔 We have seen this in Kafka before with Tiered Storage. What began as a proprietary differentiation (Confluent's Tiered Storage) eventually became foundational infra for Kafka (KIP-405). Being deliberate in absorbing market-proven features also helps maintain Kafka's dominant network effect at a reduced level of risk. Ultimately, this adoption pattern ends up as a "rising tide lifts all boats" scenario, because it ensures the Kafka vendors stay relevant for longer too. On another good note, Kafka development is pulsating harder than ever: • 4.0 released recently • KRaft went GA, replacing ZooKeeper • Tiered Storage went GA • The new consumer group protocol was released • two-phase commit transactions are being worked on • queues are being worked on And there have been a lot of interesting proposals being discussed - multi-tenant (Virtual) clusters, cross-cluster replication and ParKa - the marriage of Kafka and Parquet. It’s worth saying it’s still unclear whether this feature (KIP-1150) will be accepted, nor how long it’ll take. Kafka has a mixed history with this - a lot of ambitious KIPs don't go past the discussion stage. But the Aiven folks seem serious to me (the KIPs are exhaustive). It would also spell a slow death to Kafka were it to decide to NOT move in this cloud-native direction. 💀 So I'm pretty optimistic this will happen sooner rather than later!