That site took 10 seconds to load on my phone.
1: Since we are talking about hard to track down problems, I spent a week off and on tracking down a problem with the following symptoms: almost every night in the 3am hour connectivity over the mpls link would die for about 15 seconds. Turns out, the installation engineer configured the same stp priority on the new routers down there as our core router. Something must have been happening in isp network to cause recalculation in the 3am hour.
2: Eventually packet traces showed me a bug in our firewall when dhcp relay was enabled. When the client went to renew its address half way through the lease time, the server would see the unicast request and then the relay agent would send a request from each of its ips in the vlans, 2. So the server would send an ack, ack, nack. Also the client was an Avaya phone with a dhcp client that is not very robust the expiration of the lease would cause the phone to reboot.
The vendor is working on what they called cowboy code.