KIP-1150: Diskless Kafka Topics

Summary

Authors: Greg Harris, Ivan Yurchenko, Jorge Quilcate, Giuseppe Lillo, Anatolii Popov, Juha Mynttinen, Josep Prat, Filip Yonov StatusCurrent state: "Under Discussion"Discussion thread: hereJIRA: KAFKA-19161Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).MotivationDescribe the problems you are trying to solve.BackgroundThe Apache Kafka protocol has become a successful base for building streaming applications, and has attracted workloads that push the Apache Kafka implementation to new limits. The Apache Kafka implementation is designed around low-durability block storage and direct replication, and provides strong consistency and high durability backed by commodity hardware.Currently, Apache Kafka is often operated in cloud hyperscaler environments where high-reliability object storage is available and more cost-effective than block storage for equivalent workloads. The existing Tiered Storage feature (KIP-405) provides the capability to use object storage for inactive segments, and has seen widespread adoption. However, Tiered Storage does not remove the need for replication of active segments, which is the most substantial infrastructure cost for Apache Kafka operators on hyperscalers today.Multiple protocol-compatible alternatives to Apache Kafka now use object storage to fully replace direct replication and substantially lower the cost to operate a cluster on a hyperscaler cloud. These alternatives are finding market success and their adoption is rising, showing a general market interest in this optimization.Motivating QuestionShould the Apache Kafka implementation pursue an object storage optimization as is present in all alternatives?Yes, The Apache Kafka reference implementation should incorporate this innovation, and provide the capability to replace block storage with object storage.New CapabilitiesDiskless Topics allow Apache Kafka operators on hyperscalers to:Eliminate inter-zone data tr...