Broker-Side SQL Filtering with RabbitMQ Streams

Summary

RabbitMQ 4.2 introduces SQL filter expressions for streams, enabling powerful broker-side message filtering. In our benchmarks, combining SQL filters with Bloom filters achieved filtering rates of more than 4 million messages per second — in highly selective scenarios with high ingress rates. This means only the messages your consumers actually care about leave the broker, greatly reducing network traffic and client-side processing overhead. Motivation​ High-throughput event streams often deliver large volumes of data to consumers, much of which may not be relevant to them. In real systems there may be tens of thousands of subjects (event types, tenants, regions, SKUs, etc.), making a dedicated stream per subject impractical or unscalable. RabbitMQ Streams address this with broker-side filtering. Bloom filters skip entire chunks that don’t contain values of interest, while SQL Filter Expressions evaluate precise per-message predicates so only matching messages cross the network. This reduces network traffic, lowers client CPU and memory use, and keeps application code simpler. Demand for broker-side filtering is longstanding - Kafka users have requested it for years (see KAFKA-6020) — but Kafka still lacks this capability. RabbitMQ’s Bloom + SQL filtering makes selective consumption practical at scale today. Let’s walk through a hands-on example. Running the Sample App​ To run this example in your environment: Start RabbitMQ with a single scheduler thread: docker run -it --rm --name rabbitmq -p 5672:5672 -e ERL_AFLAGS="+S 1" rabbitmq:4.2.0-beta.3 From the root directory of the sample application, run: mvn clean compile exec:java The sample application uses the RabbitMQ AMQP 1.0 Java Client, as SQL filter expressions are a feature of AMQP 1.0. Publishing Events​ Consider a typical e-commerce platform generating a continuous stream of customer events: product.search product.view cart.add cart.remove order.created and many others Our sample application publishes 10 mil...