Classic HN: ITAPPMONROBOT

Summary

At the turn of the 21st century, Initrode Global's server infrastructure began showing cracks. Anyone that had been in the server room could immediately tell that its growth had been organic. Rackmounted servers sat next to recommissioned workstations, with cables barely secured by cable ties. Clearly there had been some effort to clean things up a bit, but whoever put forth that effort gave up halfway through. It wasn't pretty, but it worked for years. As time passed, though, a proprietary gateway server to communicate with credit processing agencies would crash more and more frequently. And these were bad crashes, too — the kind of crashes where the server wouldn't respond to ping and would have to be restarted manually. It wasn't really a big deal for the admin, Erik, to hit the restart button on the server when he was there, but that was only 40 hours a week. The credit union needed it to be active 24/7, but was unwilling to hire 24 hour staff in the datacenter. The problem kept getting worse and worse, so the IT manager called up a meeting. "OK guys, what can we do about this?" asked Laura, the IT manager. "Can you guys in dev fix this?" "No," began Erik, before anyone in dev could respond. "The issue is with the server, not our software." "Well, when does the support contract end?" "Two years ago." "Great. And we can't replace the unit while we're in a budget freeze..." Laura wasn't sure what to do. "Well, what's our workaround for now? What happens when it goes down?" "Right now, I just hit the restart button." "OK, well, we'll have to replace it once I get the budget approved. For now, though, what can we do? We need this online all the time." Laura sighed and began tapping her pen on the table. "No one has any other ideas?" At this point the room fell silent and everyone tried to avoid making eye contact with Laura. Erik had a script running that would ping the server every few minutes and alert him if it didn't respond so he could halfway proactively keep ...