Halloween Horrors: Real-Life Scary Stories from the Data Center and Beyond

The end user’s ghost haunts us all with innocent mishaps this Hallows Eve. From security oversights to trying to troubleshoot their own issues or accidentally breaking equipment with an unfortunate potion of liquids and electricity, IT pros have seen it all. With the spookiest day of the year approaching, SolarWinds polled its THWACK^® community to see what real-life scary stories community members could share from the data center and beyond.

IT professionals have only one hope for a less frightening holiday next year—end-user education. The more prepared users are when it comes to safety and security practices, the less technology professionals will be haunted in the months to come.

Here are the six scariest, most goose bump-inducing stories SolarWinds heard from its THWACK community members.

A haunted campfire tale

“About ten years ago we had a fire in the electrical switching room of our data center. Of course, it destroyed the entire building. The recovery process was so challenging some managers brought in sleeping bags and camp beds to get some sleep before starting the next shift.”

User: David Botfield, Network Engineer

“We had an air conditioner that could do it all in the room next to the Network Operations Center – heating, cooling, dehumidifying, and particulate removal. One day it caught fire, leading to chaos. The data center fire suppression was triggered by the smoke, staff had to evacuate, and we came close to losing the data center.”

User: jm_Sysadmin, Senior Systems Engineer

A guy with an axe

“I was working at a Tier 1 data center in downtown Cincinnati that was under construction. A worker wielding an axe accidently chopped right into one of the risers that carries the cooling solution up and down the building. Not knowing what to do when things were heating up in the middle of the night, they had to kill the power completely. It took weeks to recover from that mistake.”

User: ecklerwr1, Network/Systems Engineer

“A worker with a reciprocating saw inadvertently cut through all fibers in a bundle on the other side of a wall. Since then, we use diverse physical paths and require geographic diversity for all entry points to network rooms, point of presence rooms, and data centers.”

User: Rschroeder, Network Analyst

No one is safe

“A few months ago, I got a call from a supervisor saying their computer kept popping up a message and wouldn’t go away. The pop-up was our application whitelist asking for administrative credentials to run an application. Soon the whole story was uncovered. An employee in the purchasing department received an email forwarded from our security officer, which was allegedly from UPS, sending a package delivery notification. Of course, being in the purchasing department, this is very routine, so he clicked the link. Now we spend a lot of time and resources on security training for our staff!”

User: Shuckyshark

A stormy night

“Last summer we had a thunderstorm knock out the power in our building. This wasn’t a problem; the data center has battery backup and a generator. The UPS power kicked in immediately, and the generator was running fine. After about three hours, the generator had a seal fail and all the coolant leaked into our parking lot. After it overheated, our data center ran about ten minutes on the remaining UPS batteries before it went dark for about 24 hours. Imagine the trouble we found when power was finally restored!”

User: NickZourdos, Network Administrator

A nightmare-ish game

“At a previous place of work, a former IT director regularly woke up the on-call Network Analyst at 2 a.m. to complain about slowness to his home’s remote connection to work. He had a T1 in his home provided by the employer so he could work on tasks outside of normal hours. His complaints weren’t related to work, however; he was addicted to an online game and believed he was losing to players who had lower latency between their home PC and the gaming server. This went on for months. He was eventually released from the company.”

User: Rschroeder, Network Analyst

Never push that button

“Ten years ago, our electricians were reworking the Emergency Power Off options in two of our data centers. The switches were installed in each, but didn’t have protective plastic covers on them. When one of the electricians was finished using a folding step ladder he collapsed it and set it up against the exact emergency button he had finished installing and shut down our entire primary data center.”

User: Rschroeder, Network Analyst