As Robert burns famously wrote "The best laid plans of mice and men oft go astray." As we moved the lab from SUNY to our new digs in beautiful New Jersey a series of failures convinced me that he was right, and that Edsel Murphy PhD had come to visit. First, we discovered that the power supply in one of our key VMware hosts, a Dell PowerEdge 2950, didn't survive the trip. Since the lab isn't a production environment, we didn't equip it with redundant power supplies, which turned out to be a bigger mistake than we would have believed as both of the replacements I ordered from eBay turned out to be duds. But wait. . .there's more.
Then the gremlins started in on the Internet connections. The lab shares a building, and internet connections, with a medical office management company. Before we moved in, they had a 70mbps point-to-point wireless link and a Sprint T-1 for backup. We ordered a Verizon FIOS line when we moved in. As our little story opens, Verizon had come to install the FIOS, but we hadn't actually integrated it into the link load-balancer or even tried to use it.
On Tuesday, the wireless link went down and the provider expected it to stay down for a week or so. Thursday, I went to the lab and found a tech from the medical folks' VAR working in the telco room, which should have made an old network hand like me nervous.
A few minutes later, my Pandora stream died as the Sprint line bit the proverbial dust. Turns out the battery backup for the telco mux had failed rather spectacularly actually, with smoke and leaking acid. The telco stopped supporting that backup system, so the VAR had ordered replacement batteries and our friend the tech blew up the mux by changing them without disconnecting the backup power from the system.
No problem, I said, we'll just use the FIOS line. But the FIOS line was only configured to supply one IP address and even that was down. Strike three!
