Slow site performance?

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
That's good feedback. I still haven't determined what the issues have been for a few of our members the other day. Maybe we should collaborate on random performance issues.
 

Stereodude

Not really a
Joined
Jan 22, 2002
Messages
10,865
Location
Michigan
I had a few more of the same sort of hiccups today as well. I was on a Verizon LTE mifi (again) and I found one or two other sites were equally slow, but everything else was quick.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
That sounds like it might be something on Verizon's end if you're noticing other sites being equally slow? I looked through the performance logs for today and I didn't see any abnormalities. They also don't paint the entire picture so it's possible there might be network issues happening.
 

Stereodude

Not really a
Joined
Jan 22, 2002
Messages
10,865
Location
Michigan
That sounds like it might be something on Verizon's end if you're noticing other sites being equally slow? I looked through the performance logs for today and I didn't see any abnormalities. They also don't paint the entire picture so it's possible there might be network issues happening.
Maybe, I only really noticed one site that was also unloadable around the same time the forum was "pausing" for several minutes when I would click on a link.

This: http://www.fox.com/gotham

I only mention that because it was the first other site I found that wouldn't load. I don't recall if there was another.
 

timwhit

Hairy Aussie
Joined
Jan 23, 2002
Messages
5,278
Location
Chicago, IL
That Fox site is slow for me on Comcast Business internet. I think we have at least 100Mbsp downstream.
 

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
16,624
Location
USA
The site was slow for me yesterday 12pm MDT on the mobile.
 

Howell

Storage? I am Storage!
Joined
Feb 24, 2003
Messages
4,740
Location
Chattanooga, TN
That site took 10 seconds to load on my phone.

1: Since we are talking about hard to track down problems, I spent a week off and on tracking down a problem with the following symptoms: almost every night in the 3am hour connectivity over the mpls link would die for about 15 seconds. Turns out, the installation engineer configured the same stp priority on the new routers down there as our core router. Something must have been happening in isp network to cause recalculation in the 3am hour.

2: Eventually packet traces showed me a bug in our firewall when dhcp relay was enabled. When the client went to renew its address half way through the lease time, the server would see the unicast request and then the relay agent would send a request from each of its ips in the vlans, 2. So the server would send an ack, ack, nack. Also the client was an Avaya phone with a dhcp client that is not very robust the expiration of the lease would cause the phone to reboot.

The vendor is working on what they called cowboy code.
 

snowhiker

Storage Freak Apprentice
Joined
Jul 5, 2007
Messages
1,668
Site was slow today for me. 10-20 secs to load site links. 4pm AZ time.

edit: up to 2 minutes now.



LOL sorry for the huge image capture size.
 
Last edited:

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
The site was down. There were some issues with the capacity configured for apache. I re-tuned it and restarted the server. The server was also at 100% swap usage and close to 100% memory usage. I think the balance between MySQL and apache was a little off.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
Turns out to be a small DoS attack is hammering the server today bringing down the various services due to limited memory. I've put some changes in place to mitigate it for now but I'll need to do further investigation on this. Hopefully this stops the issue for now. I'll keep an eye on it.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
I may have. It wasn't a DoS against SF's domain, it was against another I host. It's originating from a single IP address at the moment which looks to be from Romania. I've since blocked it at the firewall for now to see if they come back to do more later on different IPs.
 

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
16,624
Location
USA
The site was completely down on Tuesday for a while. I thought maybe we were out of business. :(

Maybe a Romanian ex-GF was involved. ;)
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
It's been a couple of months now since the last performance-related concern. Are things still performing decently for most people? I've not encountered any issues for some time now but want to hear from others.
 

snowhiker

Storage Freak Apprentice
Joined
Jul 5, 2007
Messages
1,668
The site is still super quick and responsive.

Only problem is this.

It rare. Only once a month or so. Probably a bug somewhere in the software. I've been trained to copy my long replies to notepad so I don't lose those long replies.

Again, thanks for the site, as well as its operation, maintenance, and upkeep.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA

We've had some outages in the past few days because of these DDoS attacks. Our server is on the Newark node in case the site goes down and you want to get a status update on it from here. it sucks for them but I'm thinking the silver lining is that whatever group is coordinating the attack(s) is providing a free amount of distributed load that Linode can now use to harden their infrastructure.
 

mubs

Storage? I am Storage!
Joined
Nov 22, 2002
Messages
4,908
Location
Somewhere in time.
Site is reasonably fast now. This afternoon and morning (about 4 and 8 hours ago), it was unusable - didn't load at all. Last few days it's been dog slow. Just FYI.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
Site is reasonably fast now. This afternoon and morning (about 4 and 8 hours ago), it was unusable - didn't load at all. Last few days it's been dog slow. Just FYI.

I wasn't aware of the performance issues from the past few days but you're correct about the issues last night/early morning. I got a notification of the site being down around 2:00AM EDT. I logged in and it wasn't due to our server but rather the upstream network and routing from Linode. If you're ever interested, bookmark this page for the Linode status and look for alerts at Newark. None the less, thanks for letting me know you noticed an issue. If you're noticing performance issues, please feel free to mention it. I don't get upset by it and would rather know you're having issues so I can look into them rather than you have a frustrating experience.

I was looking at my stats and history from uptimerobot monitoring the other night and had a shake my head moment because my self-hosted website had a lot less incidents of downtime when compared to the one hosted on Linode. I realize it's not enough data to warrant that kind of change as the infrastructure I have has too many failure points...but with that said, Linode should have lots of redundancy and they still have downtime.
 

mubs

Storage? I am Storage!
Joined
Nov 22, 2002
Messages
4,908
Location
Somewhere in time.
I think apart from your hosting provider, there are network issues. Many foreign sites were slow to dead for me yesterday - like ARS Techinca, for example. There's something screwy going on. Maybe a major DOS attack somewhere, or a network outage.
 

Stereodude

Not really a
Joined
Jan 22, 2002
Messages
10,865
Location
Michigan
About half the time when I was in China the site wouldn't even load. Other times it worked fine. I didn't bother to mention it because, well, China...
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
Are we running new software? On a new server? Storage Forum has always run well for me but it seems especially nippy now. Nice work Doug!

Thanks Tea. We're still using the same software here (vbulletin 4.x). The server is somewhat new in the past several months. It's one of those new cloud services all the kids rave about these days.
 

Howell

Storage? I am Storage!
Joined
Feb 24, 2003
Messages
4,740
Location
Chattanooga, TN
This site has been very slow for the last hour. Slow like I've never seen it. Seems to have recovered now though.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
All this talk made it a self conscious cloud. I haven't seen any alerts from my variety of checks but I'll take a look into the performance issues.
 
Top