Pipelined State Machine Corruption

Summary

There are a number of network protocols that we might refer to as text protocols, where we send a line of text to the server and it sends back a response. Perhaps I should call them CRLF protocols, to distinguish from protocols where we’re blasting JSON back and forth. To speed things up, it can be tempting to have the client send multiple requests without waiting for individual responses.This is called pipelining. Support varies by protocol. NNTP requires that servers support it. HTTP 1.1 defines it, but support is effectively dead because. SMTP offers it as an option, but otherwise requires that clients not pipeline due to server bugs. I thought this was surprising. How can this happen? In my mind, a simple server, unaware of pipelining, will read one request at a time, process it, and send the reply. The pending pipelined requests will simply wait to be read. What can go wrong?What I came up with is that the server might be relying on implicit state to drive its state machine. A server of the time would likely be using select to multiplex IO. I’m imagining a server might assume that each connection can only create one wakeup for one reason at a time. We wake up to read from the net connection. We read the MAIL FROM, then start some operation to check that, and wait for a response. If we wake up again because the net connection has RCPT TO pending, we might reply to that before the from address has been checked. This is quite some time before SPF, but we can imagine something like it.So the intended sequence goes like this:fd 3 ready read MAIL FROM alice send DNS request fd 4 ready read TXT record write 250 OK fd 3 ready read RCPT TO bobby write 550 nobody home fd 3 ready read RCPT TO carol write 250 OKBut if fd 3 is ready too soon, we might get mixed up.fd 3 ready read MAIL FROM alice send DNS request fd 3 ready read RCPT TO bobby write 550 nobody home fd 3 ready read RCPT TO carol write 250 OK fd 4 ready read TXT record write 250 OKSo now we rejected the MAIL FR...