What If We Could Rebuild Kafka from Scratch?

Summary

Do away with partitions: topic partitions were crucial for scaling purposes when data was stored on node-local disks, but they are not required when storing data on effectively infinitely large object storage in the cloud. While partitions also provide ordering guarantees, this never struck me as overly useful from a client perspective. You either want to have global ordering of all messages on a given topic, or (more commonly) ordering of all messages with the same key. In contrast, defined ordering of otherwise unrelated messages whose key happens to yield the same partition after hashing isn’t that valuable, so there’s not much point in exposing partitions as a concept to users. Key-centric access: instead of partition-based access, efficient access and replay of all the messages with one and the same key would be desirable. Rather than coarse-grained scanning of all the records on a given topic or partition, let’s have millions of entity-level streams! Not only would this provide access exactly to the subset of data you need, it would also let you increase and decrease the number of consumers dynamically based on demand, not hitting the limits of a pre-defined partition count. Key-level streams (with guaranteed ordering) would be a perfect foundation for Event Sourcing architectures as well as actor-based and agentic systems. In addition, this approach largely solves the problem of head-of-line blocking found in partition based systems with cumulative acknowledgements: if a consumer can’t process a particular message, this will only block other messages with the same key (which oftentimes is exactly what you’d want), while all other messages are not affected. Rather than coarse-grained partitions, individual messages keys are becoming the failure domain. Topic hierarchies: available in systems like Solace, topic hierarchies promote parts of the message payload into structured path-like topic identifiers, allowing for clients to subscribe to arbitrary sub sets of...