Slow site performance?

snowhiker

Storage Freak Apprentice
Joined
Jul 5, 2007
Messages
1,668
Not a huge deal but thought I'd mention it in case there is some hidden issue.

Navigating the site around 3 A.M. AZ time (got off work early) noticed several delays of around 20 seconds to load pages. I thought it was perhaps my end but I did receive this error once:


Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at webmaster@storageforum.net to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.


I have NEVER seen that error before. Probably nothing but again thought I'd mention it.

Thanks.
 

snowhiker

Storage Freak Apprentice
Joined
Jul 5, 2007
Messages
1,668
Just a FYI.

Site seems to be running a bit slow at the moment. Like 15-30 seconds to server requests. Around the exact time as my OP in March. Daily maintenance maybe?
 

mubs

Storage? I am Storage!
Joined
Nov 22, 2002
Messages
4,908
Location
Somewhere in time.
It has been excruciatingly slow for me whenever I visit SF anytime of the day. I'm sure Handy can check the logs to see what times I have visited.

Something is definitely wrong.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
Sorry that you're experiencing poor performance. I'll take a look into it.
 

Bozo

Storage? I am Storage!
Joined
Feb 12, 2002
Messages
4,396
Location
Twilight Zone
It might be an issue in his local area. Here in eastern Pa the site is working fine.
 

mubs

Storage? I am Storage!
Joined
Nov 22, 2002
Messages
4,908
Location
Somewhere in time.
Much faster today.

I haven't experienced slowdowns with any other sites visit: ASR Technica, News.com, etc.
 

snowhiker

Storage Freak Apprentice
Joined
Jul 5, 2007
Messages
1,668
Been having random 5-10 second delays the last 3-5 days when navigating the board/site. Don't know if it's my end or not. I'm on 100/10 mb/s cable.

I just had a site hiccup and thought I should post this. Took about 45 seconds after I clicked POST and got the "Internal Server Error 500" or something like that. Hit BACK then clicked POST again and got the "this msg is a dup" message. So my post posted but just wanted to throw you an FYI.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
Thanks for the feedback. I'll look into the server and see if there's some performance problems happening.
 

Chewy509

Wotty wot wot.
Joined
Nov 8, 2006
Messages
3,327
Location
Gold Coast Hinterland, Australia
FYI, just got:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator at webmaster@storageforum.net to inform them of the time this error occurred, and the actions you performed just before this error.
More information about this error may be available in the server error log.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
11-Dec-2014 12:36 +10:00

Had this happen yesterday once, but have had it 2 times today...
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
Thanks Chewy for the additional data points.

My initial investigation reminded me that I left the vbulletin 5.1 instance available for people when the upgrade situation didn't work out (and I forgot to clean it up). I get notifications on my admin email when there are database and connectivity issues and there were signs pointing to that existing instance as a potential culprit contributing to the health of the server. I've removed it and cleaned up the files and leftover database. This may not be the entire reason but it does consume extra resources when the pages are indexed and accessed. Now they're all removed.

I'm currently doing some other back-end upgrades right now so it may seem slower for a an hour or so. Once that's done I plan to bounce Apache and possible MySQL to give it a fresh start hoping that this might speed things up. It's been up for a while now and may benefit from a process restart.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
Yeah...sadly. I've done some restarts of Apache, MySQL, and a few other processes. Please let me know if you're still having intermittent outages. If so I'll open a support ticket and see if something else if happening possibly on the network level.
 

snowhiker

Storage Freak Apprentice
Joined
Jul 5, 2007
Messages
1,668
Another FYI.

Took about 45 seconds to complete a post @ 3:25am (AZ time) seems like it took a while to post.

If this sounds like complaining. I'm not. Completely happy with this board, its users and its operations, just concerned that things haven't been "right" this past week or so.

MUCH THANKS.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
I'm not taking it as complaints; don't stress. I want to know if the site is performing bad so I can work on fixing the issue. I'll raise a support ticket with the hosting company and see if they can help me track down the issue. The host should in theory have ample resources. I'll check and see if there are backups scripted on the hosting company's level near that time. Some of the other backup scripts that I maintain don't run near that window of time. If there is a slowdown near 3:00AM EST then I might say it's part of my backups, specifically for the database.
 

Stereodude

Not really a
Joined
Jan 22, 2002
Messages
10,865
Location
Michigan
FWIW, I was getting 500 internal server errors this morning just after 7am eastern time. Refreshing generally worked, but it was still quite slow. All seems fine now.
 

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
16,624
Location
USA
It's been dying off for me several times over the past few days. The speed is fine now.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
I tried a couple more things to see if performance would improve and to see if this would reduce issues with errors. Can you let me know if you're still seeing slow performance, timeouts, and/or errors?
 

snowhiker

Storage Freak Apprentice
Joined
Jul 5, 2007
Messages
1,668
The site is fine this morning. :)

+1

Site is nice and zippy now. Less than 2 seconds to load a new/different thread, launch a post, preview a post or just general site navigation. Was taking 5-10 seconds for loads before.

Post speed is better also.
<edit: 2 seconds to post vs 45 secs before>

Much thanks.
 

snowhiker

Storage Freak Apprentice
Joined
Jul 5, 2007
Messages
1,668
I forgot to post this when it happened but site was really slow again and I got an "Internal Server Error" again. I think around 3'ish am AZ time, but not sure. Sorry don't have specifics so not much help.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
I forgot to post this when it happened but site was really slow again and I got an "Internal Server Error" again. I think around 3'ish am AZ time, but not sure. Sorry don't have specifics so not much help.

I'll have to put in a ticket since we're still seeing intermittent internal server errors. During that time I have nothing scheduled so you're past my backup window but the hosting company does their own backups and maybe it's in their window. I'll check into that. I'll also see about moving us to a different server. They now offer servers with SSD-based storage that maybe could help here but I'm surprised that we see this many performance issues given how small our traffic is. I have been (recently) logging MySQL slow queries to see if there is an issue going on there. I need to spend some time and look through the logs again.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
I've had a ticket open for the past several days regarding the poor performance and Internal Server Errors that have been showing up. I don't quite have a root-cause for this but I am working on trying to figure out what the issue is.
 

CougTek

Serial computer killer
Joined
Jan 21, 2002
Messages
8,724
Location
Québec, Québec
I haven't noticed anything regarding slow access to the site, but you're the one who sees the logs, so if you say there are error messages, then there must be.
 

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
16,624
Location
USA
I haven't noticed anything regarding slow access to the site, but you're the one who sees the logs, so if you say there are error messages, then there must be.

SR response was horrible over the past few days. There were timeouts and very slow response. I can't remember the times, but it was not just a single short period.
I figured there was something occurring on the server because the Google was normal.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
My apologies for some site interruptions this evening. I've been doing some Apache and PHP recompiling to incorporate some changes which I hope will improve the site's performance a bit. I subjectively feel like it's already a bit faster. I'm now adding in xcache to provide some PHP caching abilities. There may be a few more blips while I sort some of this out.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
I think I'm done tuning for tonight. The site feels a shit-ton faster than it has been in a long time. If you observe otherwise, please let me know. Also if you notice anything strange, also please let me know so I can look into it.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
It might seem a little slower for the next half hour or so. I'm extending the backup script and working on testing/debugging it. I've enabled it to securely backup automatically to my home Linux VM.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
It's possible. I wasn't on much last night so I couldn't say first-hand. It seems ok now, how is it on your end of things?
 

Stereodude

Not really a
Joined
Jan 22, 2002
Messages
10,865
Location
Michigan
Well, the honeymoon is over... It took me over 7 minutes to get into this thread so I could complain that the forum is super slow. :(

3:35-3:43 EST...

Edit: and pretty much right after posting this the forum is flying again... :scratch:
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
Well, the honeymoon is over... It took me over 7 minutes to get into this thread so I could complain that the forum is super slow. :(

3:35-3:43 EST...

Edit: and pretty much right after posting this the forum is flying again... :scratch:

I appreciate you letting me know with the date/time. I'll check my logs later tonight to see if I can find some event related to this happening. It's possible they had an interruption in service via their network. I'll check their support for that also.
 

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
16,624
Location
USA
I appreciate you letting me know with the date/time. I'll check my logs later tonight to see if I can find some event related to this happening. It's possible they had an interruption in service via their network. I'll check their support for that also.

I received a not found a couple times, and then the site was back again.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
I've started looking into this and so far I'm not seeing signs of issues related to CPU or disk. There have been no errors reported from the forum for connectivity issues. The mysql.slow log has no slow queries reported which would indicated host performance issues which thereby cause the database to run slow.

This is what I'm seeing in the past 24 hours. I've gone through all the various logs and I'm not seeing any errors (yet). I'll keep investigating. If you continue to see outages or problems, can you please post them along with the rough time you saw the issue?

Here is our host's metrics for the past 24 hours.
linode944698_cpu.png
linode944698_disk.png
linode944698_ipv4.png
 

snowhiker

Storage Freak Apprentice
Joined
Jul 5, 2007
Messages
1,668
No problems/issues for me here in the Phoenix, AZ region of the country.

Thanks for the upgrade.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
21,564
Location
I am omnipresent
I have to say it's been beautiful so far and as somebody who spent all damned week looking for a random performance issue on a web server you have my sympathies.
 
Top