Denise Fishburne always comes prepared. When I met "Fish" in person, she was kneeling on the floor of the Moscone Center in San Francisco in Cisco Live's crowded social media lounge. A mutual acquaintance pulled me over to introduce us, and she looked up from a bag where she was sorting feathery Mardi Gras masks and tin foil tiaras. She handed me a tiara, explaining her plan to take selfies with her new friends wearing the accessories. "These are my props," said the woman renowned not only for her technical expertise, but for her talent in nurturing and mentoring coworkers. "I really don't know how to do this social media thing, so I brought some tools."
Anticipating the situation and identifying the right tool for the job are skills Fish has honed through 30 years of experience troubleshooting networks as a network engineer. A team lead in Cisco's Customer Proof of Concept Lab in Research Triangle Park, N.C., Fish helps Cisco customers with tough networking challenges and provides Cisco employees with instruction and training. She will be presenting the session Techniques of a Network Detective later this month at Interop Las Vegas.
In her session, Fish will focus on the methodology of solving network problems with deductive reasoning -- gathering the facts, eliminating the unnecessary, and zeroing in on the evidence that matters. A fan of Sherlock Holmes, CSI, and Blacklist, Fish equates every network outage to a crime scene that can be solved by examining the details closely and interpreting them based on experience. She also admits that she applies her rules of network troubleshooting to navigate many aspects of daily life.
Being a network engineer has taught Fish to analyze her environment to identify patterns that exist, she said in an interview. "You just need to pay attention, which most of us don't take the time and energy to do. It works with people, and it works if the network isn't behaving the way you normally expect it."
The key is knowing what the "sphere of normal behavior" is, she explained. For example, do your packets normally go from New York to Chicago to California, or do they normally go from New York to Florida to California? Understanding what is expected allows you to differentiate between the "facts" of usual operations and "clues" that something is happening outside that normal range. Once you have a baseline, you can identify trouble, both in networks and in life.
Troubleshooting people
"Network troubleshooting is a lot easier than people troubleshooting; because network troubling is extremely black and white and people are very gray," said Fish. But the same principles can be applied. If you know what's normal for the character and behavior of people around you, then you can observe what's not normal. A basic example is when a colleague wears a suit to work in a casual office. "You know they're interviewing for a new job. Or, if it's a black suit, they may have to go to a funeral that day," said Fish. "These clues scream; they're so obvious. But not everyone is necessarily listening to it."
More specific actions of people can be intuited, as well. Fish related that when a colleague contacts her on instant message and asks to talk live on the phone or in person, that's unusual. "If it's a networking peer, I can deduce that this person wants to tell me that they have found a new job or are looking for a new job. If it's someone on the sales engineering team, they are probably looking for favoritism on their project when it comes into the lab."
Troubleshooting your environment
A confessed "control freak," Fish uses her analytical skills to prepare for situations and adapt to her environment as well. "I try to anticipate and figure out all of the different potential scenarios that might happen," she said. A simple case is when she's teaching at a conference and can predict the questions people will ask.
Careful observation is also especially useful when traveling, and can be crucial in a foreign country. "We learn to troubleshoot people in our own cultural environment, and we assume those lessons will transfer to other people and their environment. But that's not always true," Fish said.
She told about a trip to Japan, where, among other cultural differences, she noticed a distinctive "dance" among those riding in the elevator. Without speaking or making any eye contact at all, the most junior females would position themselves next to the doors to operate the open and close buttons. If there weren't two young women, then a junior male would stand on one side. "I observed this phenomenon for a week. Then the second week I was there, I joined in on the dance," she said.
Troubleshooting to prevent problems
IT pros should not only view troubleshooting as something applied after the fact, said Fish. Just as you can predict what can happen from past behavior, you can apply what you learn from troubleshooting today to prevent problems in the first place. "We live in a reactionary environment where people get 'dive and catch' rewards for solving problems that are very large. There's far too much focus on fixing problems, and a huge missed opportunity in prevention."
We can take a cue from the way jewelry stores are run, Fish noted. They use cameras, security guards, a surveillance system, and have trained their employees to act in very specific ways to deter theft. The protection and surveillance of packets and corporate data can benefit from similar measures.
What can go wrong in troubleshooting? Fish said the number one mistake is jumping to a conclusion without examining all the details and their context. It's tempting because everyone wants want to solve their problem as quickly as possible. But often, especially in a network, there may be more than one problem going on at once, and all of the clues may not lead to the same conclusion -- so it's important to tease out the right information. "A great network detective has to be able to let the evidence guide them," she advised. "You can have a theory, but the evidence still has to guide you."
To learn more about troubleshooting methodologies, tools, and interpreting the evidence, join Denise Fishburne for Techniques of a Network Detective at Interop Las Vegas on Wednesday, April 29, at 2:15 pm. You may even be able to take your own selfie with Fish -- tiaras are optional.
Discover the benefits of scalable backbone and network services for application and cloud providers.
Network automation is gaining momentum. Here's how you can drive this powerful technology to its full potential.
Subsea cable alternatives can provide a level of comfort for those concerned about disruptions. Realistically, they will continue to play a small, but critical role in the short term.
