VoIP Packet Fight: Troubleshooting Phone Initialization Problems

I was working on an issue with a client where his VoIP phone’s get stuck at the ‘initialization’ process. As we chatted over the phone, we quickly determined that this issue occurs when a phone is powered off and back on.  The client also mentioned that this seemed to start about a year ago when they got a new router installed.

I asked how he typically resolved the issue and he said, “I just reboot stuff until everything works.”  I have to give him credit, at least he was honest.

Just a few things I checked when I got on site:

• Yup, the problem is definitely repeatable which is a huge plus when troubleshooting.
• Verified that the phone had a link light on its Ethernet port
• Phone reports that it has an ip address assigned that we can ping
• Tested the PC pass through port with my laptop which works fine

Now I’m intrigued and need a trace file to see what’s going on. Like all POE (power over Ethernet) devices, the challenge is how to capture the packets as the phone boots up. Oh yeah, and the switch is managed by a third-party, so we don’t have the option to set up a monitor/span port. Other options would be to use an external power supply for the phone and a tap or even an old 10/100 true hub.

That’s when I reached into my bag and reached for my Profitap/ProfiShark tap, since I can connect the tap inline between the phone and the switch port.  I run into many switch configurations that have some sort of port security, so I want to avoid locking out a port. The ProfiShark connects to my computer via an USB3 port, keeping my MAC address out of the switch port and trace file as well as passing POE through the ports.

Back to the trace file. As you will see in the video, I started with getting rid of the details and Bytes panel within Wireshark to give me a better view of the data. Then I set up a display filter for DHCP packets to verify that there were no DHCP errors, and that the basic 4 step process worked fine. Keep in mind that all VoIP systems are different so review your trace closely for and other protocols/conversations.

My next step was to set up a display filter for the MAC address of the phone. That is when I saw the Router responding to a phone’s gratuitous ARP. I suggested we disconnect the router LAN port and immediately the phone got past “initializing” message and the phone was operational.

What was additionally odd in the trace is that even though the phone and gateway successfully got 192.168.0.0 ip addresses, they used 169.254.x.x addresses. 

At least now we understand why the phones got stuck on initializing and we could find a workaround to get things up and running.  Now I just must figure out the answers to these, and other questions that will surely pop up.