You don't need Kafka: Building a message queue with Unix signals

Summary

Have you ever asked yourself what if we could replace any message broker with a very simple one using only two UNIX signals? Well, I’m not surprised if you didn’t. But I did. And I want to share my journey of how I achieved it. If you want to learn about UNIX signals, binary operations the easy way, how a message broker works under the hood, and a bit of Ruby, this post is for you. And if you came here just because of the clickbait title, I apologize and invite you to keep reading. It’ll be fun, I promise. It’s all about UNIX A few days ago, I saw some discussion on the internet about how we could send messages between processes. Many people think of sockets, which are the most common way to send messages, even allowing communication across different machines and networks. Some don’t even realize that pipes are another way to send messages between processes: $ echo 'hello' | base64 aGVsbG8K Here’s what’s happening: The process echo is started with the content “hello” echo is a program that prints the message to STDOUT Through the pipe, the content in STDOUT is sent directly to the STDINT of the base64 process The base64 process encodes its input to Base64 and then puts the result in STDOUT Note the word “send”. Yes, anonymous pipes are a form of IPC (Inter-process communication). Other forms of IPC in UNIX include: named pipes (mkfifo) sockets regular files or even a simple signal UNIX signals According to Wikipedia: A UNIX signal is a standardized message sent to a program to trigger specific behaviour, such as quitting or error handling There are many signals we can send to a process, including: SIGTERM - sends a notification to the process to terminate. It can be “trapped,” which means the process can do some cleanup work before termination, like releasing OS resources and closing file descriptors SIGKILL - sends a termination signal that cannot be trapped or ignored, forcing immediate termination SIGINT - the interrupt signal, typically sent when you press Ctrl+...