Flaky network

[ddrueding]

This one is killing me. My network has ~600 devices across 5 sites. Everything works some of the time but nothing works all of the time. Sometimes the speed is fine and sometimes it takes minutes. Any network communication is effected; to your network printer, network shares, RDP sessions, even the web interface of my firewall. They all work 90% of the time but will all drop within minutes if you keep testing. There is no single piece of hardware that all these network paths go though; no single switch or firewall, and even static IPs are effected. I'm running out of places to look.

Help?

 

[time]

Packet storm? Do your routers block broadcasts?

What's the topology between the sites?

I know this sounds basic, but what's the longest segment cable length and how many segments are switched end-to-end?

Do you have the same switches and routers on every site?

 

[CougTek]

A network storm would be my guess too. Are you sure your main switches have a sufficient switching capacity for your traffic? Maybe you haven't segmented your network enough through the use of VLAN?

I'm curious to read what will be Chewy's view on this.

 

[ddrueding]

Mixed switches, and a completely open topology. Traffic can freely broadcast across the whole system including wireless WAN links, fiber connections, etc.

Going against some best practices, sure, but it worked flawlessly until this morning when all hell broke loose. This started at the same time as our phone system went ballistic, but disconnecting the phone system from the data network did not resolve the issue.

 

[Howell]

What does your network monitoring system tell you?

 

[CougTek]

Mixed switches, and a completely open topology. Traffic can freely broadcast across the whole system including wireless WAN links, fiber connections, etc.

You said that you had ~600 devices? And they can all broadcast all over the network?


There's a reason to follow best practices.

Maybe you can try this application?

 

[ddrueding]

Finally got through the wireshark traces, and I'm pretty sure I've found the problem. Looks like the phone vendor put in several bridges between his network (VoIP, restricted to specific ports in this building, managed by him) and my network. He has a Linksys router handling DHCP for all the phones, with certain ports forwarded from my network for management, etc. As our networks never touch other than his WAN port, I've let him manage it as he likes. Looks like one of his installers bridged his network to mine in at least three locations (that I've found) today. So not only did I have a rouge DHCP server handing out IPs in a range that conflicts with the class C that my servers are on, but a colossal amount of traffic.

What a rough day.

 

[Howell]

Man that sucks! As if there are not enough gremlins to fight without vendors letting more loose.

 

[ddrueding]

Yup. Having two incompatible networks in close proximity is a risk, but I had made troubleshooting easy by using only blue patch cables for data and white for voice, with red for uplinks to other switches. Of course, when I got in when he was done they were all over the place and included orange, green, and pink. Up til 4AM getting it sorted out.

 

[Mercutio]

In theory, this is where having managed, locked down switches would've been super helpful. You could've required 802.1x port authentication and defaulted their crap to a null VLAN or something and then at least they'd have to involve you at the point when they're plugging crap in to your ports.

 

[ddrueding]

In theory you are 100% correct. But one night of troubleshooting every 7 years is worth not having to deal with that crap the rest of the time, IMO.

 

[Mercutio]

It's not a bad back burner project. Speaking of, do you actually have your network documented and diagrammed, or is it all in your head? Do you have anything that vendors can refer to for future reference?

I've done that for all my customers but it occurred to me after I read this post that I have never bothered to do it for any of the sites I maintain for my salaried job.

 

[ddrueding]

Nope, no documentation here. That was the first of my projects to be cut. When the job grew to the point that projects weren't making deadlines, I began to maintain a list of outstanding issues and projects with an importance scale of 0-9 that the boss maintains. When I feel it is worth the debate, I can get some things moved up or down the list. I hate documentation so much that I haven't bothered to advocate for it.

Of course, all my consulting clients have full documentation. I consider my liability much higher in that kind of environment.

 

[ddrueding]

On a happier note, I was cleaning up the cabling this afternoon and found a way to pull another 30% of internet bandwidth into the main pool. 540kB/s! Between 300 users! Woo!