Broken Server 2008?

[Mercutio]

I've got a 2008 Server machine. Well, actually, I have five of them.
But two of them are really, really, really slow. Slow beyond reason, given their hardware specs.

These boxes aren't really DOING anything. At the moment they're domain controllers and they have a couple file shares. I've even re-built them a couple times with different hard drives.

These machines are Q6600s with 8GB RAM on Intel 965-based motherboards.

How slow are they, you might ask?

Well, they take 36 seconds to start Internet Explorer and load "About:Blank" as their home page. Four CPUs, mind you, to do that. Approximately the same amount of time to load MSN.com.

20 seconds to display Task Manager.

Two minutes to load Active Directory Users and Computers, for a domain with a TOTAL of 18 user and computer accounts and no defined GPOs.

CPU utilization skips all over the place, but for "idle" machines I'm seeing 50% - 75% CPU utilization on all four cores for... nothing. The only running process is Server Manager. Process Explorer isn't showing me anything else using CPU time, either.

There's no sharepoint, no database, no Exchange, no nothing on these boxes except 2GB of MS Office files in a file share. I've formatted their main drives at least once. No go.

My other three servers, including my Thinkpad notebook and a couple AMD-based test boxes, are fine. The only ones giving me trouble are these Intel 965 guys.

They're on the most recent BIOS update they can have.
There's no viruses and no malware present.

No services installed but file sharing.

What. The. Hell?

 

[ddrueding]

Just to make sure it isn't hardware, what happens when you boot to an Ubuntu LiveCD? I don't think it is hardware, but good to rule it out. After that, are there any peripherals you can detach/unload? With a new OS their might be major driver issues?

 

[Mercutio]

I'm not physically in a place where I could install a DVD drive or access their USB ports at the moment. They're completely headless machines. They use Intel DG965OTMKR motherboards. I've tried Crucial and Corsair RAM. They've had Samsung and Seagate "AS" 750GB drives (slow crappy drives, I thought they were the issue at first) in them.

Essentially, I can't imagine how this hardware + this OS could be as bad as it is. My T61 has fundamentally the same hardware (Intel Core 2 Duo, 4GB RAM, G965 chipset, Samsung drive) and it flies.

Even if I boot off a Live Linux Distro, that's not going to tell me much; it's going to be sluggish for being a Live Distro, and I already know all the hardware is actually functional.

 

[sdbardwick]

You've most likely thought of this, but could the CPUs be hitting the thermal throttling temp? I've had more than one properly installed push-pin type HSF (including Intel retail) spontaneously pop one (or two!) pin(s) out and cause the CPU to throttle. [I think it is a combination of borderline manufacturing tolerances on the pins and MB, coupled with thermal expansion/contraction on those systems.]

I've also seen HSFs so clogged with dust/grime that they would cause the CPU to throttle.

If you have remote access, you can install temp. monitoring SW (Real Temp seems to be reasonably accurate) for a quick check.

 

[Mercutio]

Well, after a 20 minute adventure of opening the page and getting the program downloaded and running on both machines, I see temps that are in the normal range of 42 - 50C per core (the delay was literally waiting for the pages to load, not downloading).

In a server room. Both machines are in 4U rack cases with four physical hard disks and are apparently doing, er, something, because they both show ~50% CPU utilization for no reason that I can see.

The CPUs are well ventilated.

I've taken to using an XP box to run a counter log of CPU-related information, but it's not been very illuminating. My interrupts/second (usually an indicator of bad hardware that would cause processor utilization to spike) appear to be normal. The sad thing is that I can remote in and run the XP machine and it's faster and more responsive than the servers. The XP machine is a 1.6GHz single-core Celeron.

 

[Stereodude]

Man these posts are so good I gotta read 'em twice!

Are you sure DMA or the equivalent for the SATA HD's is enabled on 'em?

 

[Mercutio]

I'm running the drives through ACHI. There does not appear to be a DMA mode setting in Windows and I don't remember seeing anything in the BIOS for it either.

178 minutes to copy 2GB of files from one of these machines to NAS box that's on the same 1GBit switch. Server 2008 also uses the "more accurate" file transfer info that Vista does, so I guess I'll be able to start reformatting in about five hours.

 

[Bozo]

Do you need to (or have you) updated the motherboard's INF drivers from Intel??
Systernals used to have a free utility that would break down the processes to look for unseen usage. I beleive they are on MS site now.

http://technet.microsoft.com/en-us/sysinternals/0e18b180-9b7a-4c49-8120-c47c5a693683.aspx

Bozo :joker:

 

[Bozo]

It wouldn't be some like the Defrag program running in the backround?
It is automated and enabled in Vista, but I don't remember if it was in Server 2008.

Bozo :joker:

 

[Mercutio]

Drivers are off the Intel web site, current as of Sunday.

I've been looking at Process Explorer already. Again, nothing is REPORTING that it's using massive CPU time, but the CPU time is reporting being used. It was like that before the previous wipes and the hard disk change as well.

At this point I think I can either replace both motherboards or I can switch the Server 2008 licenses to Server 2003. I haven't decided which I'll do, but these machines are so intolerably awful I can't stand to have them any longer.

 

[Stereodude]

178 minutes to copy 2GB of files from one of these machines to NAS box that's on the same 1GBit switch. Server 2008 also uses the "more accurate" file transfer info that Vista does, so I guess I'll be able to start reformatting in about five hours.

The machines sound like they have some disk IO subsystem problem, or some sort of drive encryption that makes 'em really slow. My work laptop was a speed demon prior to getting PGP Desktop's drive encryption put on it. Now it's a complete and total dog.

 

[Mercutio]

They have one standard 750GB SATA drive that's being used for the OS and 3 750GB drives that are completely unconfigured. All are on the motherboard's onboard disk controller (the three drives will be run in softRAID 5, but as I said they aren't even formatted).

And there's no disk encryption. This was a new setup.

The only thing I can think is that there's some kind of funky BIOS or driver issue with that particular mainboard and Server 2008. I *have* to get these machines running, so I'm dumping the boards for DG31PRs (which can only handle 4GB RAM) and at the same time reverting to Server 2003.

And hey, only 73 more minutes until my file copy is done!

 

[mubs]

I'm dumping the boards for DG31PRs (which can only handle 4GB RAM) and at the same time reverting to Server 2003.

If you do both together, we'll never find out what the problem was, and I most likely will die of curiosity.

 

[Bozo]

If you do both together, we'll never find out what the problem was, and I most likely will die of curiosity.

Agreed, along with the rest of us.

Bozo :joker:

 

[Mercutio]

I switched the hardware out and took the boards. I may play with them on my own. I can't believe they'd be that bad or that they'd be bad in the way that they were.

My client was on the verge of strangling me over the performance of those machines. I have about 20 non-billable hours above and beyond what I was supposed to. I think I'm going to wait a little while to tell him about the change in the amount of RAM the systems have, too. I just didn't have any choice.


Server 2003 + $65 Intel boards + all the other hardware as it was = the performance I expected to see. I guess that means the boards were the source of the problem but that means they're both bad in some weird way.

One thing I am absolutely thankful for right now is that Server 2008 gives you 60 days to activate, so I didn't waste those licenses by tying them to that hardware.

 

[LiamC]

One thing I am absolutely thankful for right now is that Server 2008 gives you 60 days to activate, so I didn't waste those licenses by tying them to that hardware.

And this is the downside of the whole activation model. If the person wasn't as switched on as you Merc, the license would have been "wasted". And if I'd "wasted" the $$$ for a server license, I'd be mighty pissed-off.

The flip side is that if it is easy to ring Microsft and explain what happened and get a new license key, then why can't pirates? The system seems to cause grief for legit people and won't deter those it's meant to.

 

[ddrueding]

It is and they do.

 

[sechs]

Were you getting a lot of kernel time?

I might suggest that AHCI had something to do with it... but then again, this is storageforum, and storage seems the obvious choice for all problems and solutions.

 

[sechs]

Were you getting a lot of kernel time?

I might suggest that AHCI had something to do with it... but then again, this is storageforum, and storage seems the obvious choice for all problems and solutions.

 

[Bozo]

One of the things I have notice about server operating systems is the way they hang when making network connections. It seems to take forever. And if you have a network card that is not connected but enabled, you can't do anything with the server for a very long time. If you try to, it takes even longer for the system to become responsive.
Motherboard mounted NICs seem to be the worst.

Bozo :joker: