Slow site performance?

[snowhiker]

Not a huge deal but thought I'd mention it in case there is some hidden issue.

Navigating the site around 3 A.M. AZ time (got off work early) noticed several delays of around 20 seconds to load pages. I thought it was perhaps my end but I did receive this error once:


Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at webmaster@storageforum.net to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.


I have NEVER seen that error before. Probably nothing but again thought I'd mention it.

Thanks.

 

[CougTek]

I can confirm that I experience the delays too and it is now ~7am on the East Coast.

 

[snowhiker]

Just a FYI.

Site seems to be running a bit slow at the moment. Like 15-30 seconds to server requests. Around the exact time as my OP in March. Daily maintenance maybe?

 

[mubs]

It has been excruciatingly slow for me whenever I visit SF anytime of the day. I'm sure Handy can check the logs to see what times I have visited.

Something is definitely wrong.

 

[Handruin]

Sorry that you're experiencing poor performance. I'll take a look into it.

 

[Bozo]

It might be an issue in his local area. Here in eastern Pa the site is working fine.

 

[mubs]

Much faster today.

I haven't experienced slowdowns with any other sites visit: ASR Technica, News.com, etc.

 

[snowhiker]

Been having random 5-10 second delays the last 3-5 days when navigating the board/site. Don't know if it's my end or not. I'm on 100/10 mb/s cable.

I just had a site hiccup and thought I should post this. Took about 45 seconds after I clicked POST and got the "Internal Server Error 500" or something like that. Hit BACK then clicked POST again and got the "this msg is a dup" message. So my post posted but just wanted to throw you an FYI.

 

[Handruin]

Thanks for the feedback. I'll look into the server and see if there's some performance problems happening.

 

[Chewy509]

FYI, just got:

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator at webmaster@storageforum.net to inform them of the time this error occurred, and the actions you performed just before this error.
More information about this error may be available in the server error log.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

11-Dec-2014 12:36 +10:00

Had this happen yesterday once, but have had it 2 times today...

 

[Handruin]

Thanks Chewy for the additional data points.

My initial investigation reminded me that I left the vbulletin 5.1 instance available for people when the upgrade situation didn't work out (and I forgot to clean it up). I get notifications on my admin email when there are database and connectivity issues and there were signs pointing to that existing instance as a potential culprit contributing to the health of the server. I've removed it and cleaned up the files and leftover database. This may not be the entire reason but it does consume extra resources when the pages are indexed and accessed. Now they're all removed.

I'm currently doing some other back-end upgrades right now so it may seem slower for a an hour or so. Once that's done I plan to bounce Apache and possible MySQL to give it a fresh start hoping that this might speed things up. It's been up for a while now and may benefit from a process restart.

 

[timwhit]

vb 5.1 still fucking things up even after no one is using it anymore.

 

[Handruin]

Yeah...sadly. I've done some restarts of Apache, MySQL, and a few other processes. Please let me know if you're still having intermittent outages. If so I'll open a support ticket and see if something else if happening possibly on the network level.

 

[snowhiker]

Another FYI.

Took about 45 seconds to complete a post @ 3:25am (AZ time) seems like it took a while to post.

If this sounds like complaining. I'm not. Completely happy with this board, its users and its operations, just concerned that things haven't been "right" this past week or so.

MUCH THANKS.

 

[Handruin]

I'm not taking it as complaints; don't stress. I want to know if the site is performing bad so I can work on fixing the issue. I'll raise a support ticket with the hosting company and see if they can help me track down the issue. The host should in theory have ample resources. I'll check and see if there are backups scripted on the hosting company's level near that time. Some of the other backup scripts that I maintain don't run near that window of time. If there is a slowdown near 3:00AM EST then I might say it's part of my backups, specifically for the database.

 

[Stereodude]

FWIW, I was getting 500 internal server errors this morning just after 7am eastern time. Refreshing generally worked, but it was still quite slow. All seems fine now.

 

[LunarMist]

It's been dying off for me several times over the past few days. The speed is fine now.

 

[Handruin]

I tried a couple more things to see if performance would improve and to see if this would reduce issues with errors. Can you let me know if you're still seeing slow performance, timeouts, and/or errors?

 

[LunarMist]

The site is fine this morning.

 

[snowhiker]

The site is fine this morning.

+1

Site is nice and zippy now. Less than 2 seconds to load a new/different thread, launch a post, preview a post or just general site navigation. Was taking 5-10 seconds for loads before.

Post speed is better also.
<edit: 2 seconds to post vs 45 secs before>

Much thanks.

 

[snowhiker]

I forgot to post this when it happened but site was really slow again and I got an "Internal Server Error" again. I think around 3'ish am AZ time, but not sure. Sorry don't have specifics so not much help.

 

[LunarMist]

It was very slow an hour or two ago.

 

[Handruin]

I forgot to post this when it happened but site was really slow again and I got an "Internal Server Error" again. I think around 3'ish am AZ time, but not sure. Sorry don't have specifics so not much help.

I'll have to put in a ticket since we're still seeing intermittent internal server errors. During that time I have nothing scheduled so you're past my backup window but the hosting company does their own backups and maybe it's in their window. I'll check into that. I'll also see about moving us to a different server. They now offer servers with SSD-based storage that maybe could help here but I'm surprised that we see this many performance issues given how small our traffic is. I have been (recently) logging MySQL slow queries to see if there is an issue going on there. I need to spend some time and look through the logs again.

 

[Stereodude]

Must be the North Koreans.

 

[Handruin]

I've had a ticket open for the past several days regarding the poor performance and Internal Server Errors that have been showing up. I don't quite have a root-cause for this but I am working on trying to figure out what the issue is.

 

[CougTek]

I haven't noticed anything regarding slow access to the site, but you're the one who sees the logs, so if you say there are error messages, then there must be.

 

[LunarMist]

I haven't noticed anything regarding slow access to the site, but you're the one who sees the logs, so if you say there are error messages, then there must be.

SR response was horrible over the past few days. There were timeouts and very slow response. I can't remember the times, but it was not just a single short period.
I figured there was something occurring on the server because the Google was normal.

 

[Handruin]

My apologies for some site interruptions this evening. I've been doing some Apache and PHP recompiling to incorporate some changes which I hope will improve the site's performance a bit. I subjectively feel like it's already a bit faster. I'm now adding in xcache to provide some PHP caching abilities. There may be a few more blips while I sort some of this out.

 

[Handruin]

I think I'm done tuning for tonight. The site feels a shit-ton faster than it has been in a long time. If you observe otherwise, please let me know. Also if you notice anything strange, also please let me know so I can look into it.

 

[timwhit]

Seems much faster to me.

 

[Handruin]

It might seem a little slower for the next half hour or so. I'm extending the backup script and working on testing/debugging it. I've enabled it to securely backup automatically to my home Linux VM.

 

[LunarMist]

It's slow again here. Anybody else notice?

 

[Handruin]

It's possible. I wasn't on much last night so I couldn't say first-hand. It seems ok now, how is it on your end of things?

 

[Stereodude]

Well, the honeymoon is over... It took me over 7 minutes to get into this thread so I could complain that the forum is super slow.

3:35-3:43 EST...

Edit: and pretty much right after posting this the forum is flying again... :scratch:

 

[Handruin]

Well, the honeymoon is over... It took me over 7 minutes to get into this thread so I could complain that the forum is super slow.

3:35-3:43 EST...

Edit: and pretty much right after posting this the forum is flying again... :scratch:

I appreciate you letting me know with the date/time. I'll check my logs later tonight to see if I can find some event related to this happening. It's possible they had an interruption in service via their network. I'll check their support for that also.

 

[LunarMist]

I appreciate you letting me know with the date/time. I'll check my logs later tonight to see if I can find some event related to this happening. It's possible they had an interruption in service via their network. I'll check their support for that also.

I received a not found a couple times, and then the site was back again.

 

[LunarMist]

OMG that last post took at least a minute.

 

[Handruin]

I've started looking into this and so far I'm not seeing signs of issues related to CPU or disk. There have been no errors reported from the forum for connectivity issues. The mysql.slow log has no slow queries reported which would indicated host performance issues which thereby cause the database to run slow.

This is what I'm seeing in the past 24 hours. I've gone through all the various logs and I'm not seeing any errors (yet). I'll keep investigating. If you continue to see outages or problems, can you please post them along with the rough time you saw the issue?

Here is our host's metrics for the past 24 hours.

 

[snowhiker]

No problems/issues for me here in the Phoenix, AZ region of the country.

Thanks for the upgrade.

 

[Mercutio]

I have to say it's been beautiful so far and as somebody who spent all damned week looking for a random performance issue on a web server you have my sympathies.