We’ve all been, at some point in our technology careers, in a situation where we felt like things could’ve gone completely wrong—but they didn’t. Did something break? Get lost or deleted? You know that feeling—all hope seemed lost, but like magic, everything worked out.
Of course, we can all agree: there’s no lucky tech pro, just a prepared one. But these near-miss miracles still make technology professionals raise a glass to luck itself for saving the moment. In honor of St. Patrick’s Day, there’s no better time to reflect on those instants when life threw us a curveball, but we were able to to a home run.
While dusting off our "Kiss Me, I'm Irish" shirts and preparing to celebrate, we asked the SolarWinds® THWACK® community of over 150,000 technology professionals to share their luckiest moments—from WAN sites going down, to porting a database onto an iPod (yes, really), to almost burning down the entire office. We’ve all experienced that moment of panic, but leprechauns aren’t the only ones who can find a pot of gold on the other side of the rainbow.
An Unlikely Use for an iPod
“I was working at the HQ for a mid-sized manufacturing firm when I received a phone call that two hard drives had blown on a production server in Mexico, and that I needed to come there right away to get their database and app up and running again. I brought with me an ERD Commander CD to boot off, and I was able to get an OS running. Between Dell support and I, we were able to get the volume mounted in a degraded state but enough where I could pull data.
“I was feeling really good, until I realized that I had no way to get the data from the server in Mexico to a donor server in the U.S. I couldn’t bring the server through customs, I didn't have time to deal with the crossing, the duties, etc., and I couldn’t bring the hard drives themselves as the donor server had a different controller because I was paranoid that moving them too much would degrade them further. To top it all off, the server had no CD burner and the database was too large for the 256MB USB stick I had on me. In the midst of my anguish I had a revelation: my iPod. I had set it for mass storage earlier, I still had the USB cable on me, it was small, portable, and no one would think twice about it. I immediately blanked it out, plugged it in, and to my amazement transferred all the DB files to it. I crossed back to the U.S. without issue, went right up to the donor server, and loaded the files. Even though the database initially wouldn't mount, DriveSavers was able to recover 99 percent of the data off of it, which places this squarely in the win column.”
Saved the Day
“It was the days of frame relay internet connections. I had been promoted to NetMan and one day, at noon, our connection to the internet dropped and would not reconnect. Suspecting a virus or something worse, I yanked the comms cable and went about troubleshooting. I remember I had inherited an old sniffer program (can't recall the name) so I did what I could to packet capture and see if there was something offending...but there wasn't! When I plugged the internet back in, the circuit got flooded almost immediately.
“Long story short, after several 18+ hour days and just when I thought all hope was lost and I would have needed to tear down and start from scratch, I found the culprit: a faulty dumb hub in our Customer Service area was spewing garbage on the wire. I replaced it, re-plugged the Gore-webs and all was right with the world! My boss at the time even gave me a little bonus (I think it was a MacDonald’s gift card) for my efforts.”
· User: asheppard970, LAN Support Analyst
Microwave on Fire
“Remember that episode of The Office where Ryan started the fire? I was almost that guy. One night, working my third shift, I went to the break room to warm up something. I was the only one at the data center and the mobile phone rang. I quickly set the time on the microwave and answered the phone. A few minutes later I came back, and smoke had filled the break room (it turns out five minutes is way too long to warm up some bread). I immediately freaked out as smoke started billowing into the office areas. I imagined smoke alarms going off and water spraying everywhere. In a complete panic, I pulled in the big floor fans we used when the air handlers would act up. The smoke began to subside, and no damage was done. I couldn’t believe it, but nothing had happened. If smoke had even whiffed a little bit in to the data centers, it could have been a catastrophe.”
The Twists and Turns of Life
“I went to college to be a police officer, but back in the 90s police departments weren't really hiring, especially people with no experience like myself. So, I got a job as a security guard in an office building working 4 p.m. – midnight and made friends with everyone who worked late. One person I struck up a good relationship with, Chuck, was starting a small computer consulting gig for non-profits. I'd been playing with computers since TRS-80s and the heydays of BBCs (look up Scepter of Goth). Chuck needed help and while he couldn't afford to pay me, he could get me experience. So, I would come in at 9 a.m., work for Chuck all day, change into my security uniform, and be at my desk at 4 p.m. Eventually, I got my experience and a PC support gig opened up in another office in the building. My reputation as a hard worker and someone who could get along with people had well preceded me, and the job was mine immediately. My career was off and running and has yet to stop.
“Nine years ago, I was hired by my current company and I was asked to manage a team of SAP administrators with absolutely zero SAP experience. I made it abundantly clear I had none, but I knew I could do it. They offered me the amazing opportunity and it has paid off hugely. I’m still learning SAP technology to this day.”
Lucky You!
“I worked for a State Lottery, and Eddie Tipton, who was the security director for the Multi-State Lottery Association (and recently convicted of fraud), would come and audit me on my network security. He would ask for my firewall and network equipment configs, but I never turned them over to him. Just didn't feel right. In hindsight, we were lucky – we found out he was caught red-handed. Sometimes you just have to say no.”
Quittin' Time
“I worked at a company where we had a few sites that were deployed with managed switches. After a few weeks, we noticed that some sites in charge of critical services were no longer functioning. We asked the local staff to replace the deceive but still couldn’t figure out what was going on. One day, I finally had time to check one of the returned devices and realized that the configuration was no longer there, and the device was in a default state. Bingo! One of my co-workers decided to modify the mode button on the second switch we sent out. Turns out that some employees wanted a longer break so we had to apply a policy to all switches of that model.
“From that experience, I learned to never to underestimate the personal and now I make sure all devices are locked down.”
Discover the benefits of scalable backbone and network services for application and cloud providers.
Network automation is gaining momentum. Here's how you can drive this powerful technology to its full potential.
Subsea cable alternatives can provide a level of comfort for those concerned about disruptions. Realistically, they will continue to play a small, but critical role in the short term.
