Show HN: Walrus – a Kafka alternative written in Rust

Summary

Walrus: A Distributed Message Streaming Engine Walrus is a distributed message streaming platform built on a high-performance log storage engine. It provides fault-tolerant streaming with automatic leadership rotation, segment-based partitioning, and Raft consensus for metadata coordination. Key Features: Automatic load balancing via segment-based leadership rotation via segment-based leadership rotation Fault tolerance through Raft consensus (3+ nodes) through Raft consensus (3+ nodes) Simple client protocol (connect to any node, auto-forwarding) (connect to any node, auto-forwarding) Sealed segments for historical reads from any replica for historical reads from any replica High-performance storage with io_uring on Linux Architecture System Overview Producers and consumers connect to any node (or via load balancer). The cluster automatically routes requests to the appropriate leader and manages segment rollovers for load distribution. Node Internals Each node contains four key components: Node Controller (routing and lease management), Raft Engine (consensus for metadata), Cluster Metadata (replicated state), and Bucket Storage (Walrus engine with write fencing). Core Components Node Controller Routes client requests to appropriate segment leaders Manages write leases (synced from cluster metadata every 100ms) Tracks logical offsets for rollover detection Forwards operations to remote leaders when needed Raft Engine (Octopii) Maintains Raft consensus for metadata changes only (not data!) Handles leader election and log replication Syncs metadata across all nodes via AppendEntries RPCs Cluster Metadata (Raft State Machine) Stores topic → segment → leader mappings Tracks sealed segments and their entry counts Maintains node addresses for routing Replicated identically across all nodes Storage Engine Wraps Walrus engine with lease-based write fencing Only accepts writes if node holds lease for that segment Stores actual data in WAL files on disk Serves reads from any...