Best Hard Drive Configuartion for a Development Workstation?

[jps0]

I am building a workstation for software development. The system will be compiling code for an average of 6 hours a day, every day. I would be curious to hear suggestions on the best disk arrangement for the system. The system will have 2.0GB of PC3200 RAM and running a 2.2GHz Athlon64. Yes, I know dual-Opterons would be ideal, but I am not made of money.

Anyway, I currently have the following disk arrangement in mind:
- 1 x 80GB SATA 8MB disc for the OSes (W2K and WXP-64), apps, and development environment
- 1 x 250GB SATA for the source code for the compile and general data storage.

But I am curious if I would see a gain from the following arrangement:
- 1 x 80GB SATA for the OSes and apps
- 1 x 80GB SATA partitioned for the development environment for each OS.
- 1 x 250GB SATA for the source code for the compile and general data storage.

My thinking with this arrangement is that it will limit wear on the HDDs and allow the highest throughput because each HDD has a different component for the compiles.

Now, I could RAID 0 the two 80GB HDDs but those arrays are prone to dying easily and I do not want to take the chance. Also, Raptors and SCSI are out of the question at the moment simply due to cost. For the 80GB HDDs I am thinking of Hitachi 7K80 7200rpm 80GB 8MB cache models and I already have the 250GB storage model.

Is there anyway to improve this arrangement for maximum throughput and fastest compile time? What else would you suggest?

 

[P5-133XL]

First, get yourself a dual-core Athlon64. If there are budgetary constraints then an 3800+ X2

For performance, more RAM (4GB). Everything that can be cached will speed up compile times because your machine won't have to go to the hard drives. Cached data is easily 10x faster than HD speed.

Next, Get yourself some 4GB I-Ram drives. They will operate at least twice as fast as your SATA HD's.

Don't worry about the wear on HD's from usage: Good drives are designed to be used. Rather seperate data on different drives that will be accessed simultanously so that the accesses will occur in parallel, rather than serially. Examples of this would be source code sould be on a seperate drive than your temp files which should be on a seperate drive from libraries.... Mind you that these need to be seperate drives and not seperate partitions on the same drive which won't gain any performance.

Since you are concerned with reliability -- Get at least one large drive that is used for backups of the other drives. Then regularly backup! Let me repeat that because it is important -- regularly backup!

 

[jps0]

First, get yourself a dual-core Athlon64. If there are budgetary constraints then an 3800+ X2

For performance, more RAM (4GB). Everything that can be cached will speed up compile times because your machine won't have to go to the hard drives. Cached data is easily 10x faster than HD speed.

Next, Get yourself some 4GB I-Ram drives. They will operate at least twice as fast as your SATA HD's.

Don't worry about the wear on HD's from usage: Good drives are designed to be used. Rather seperate data on different drives that will be accessed simultanously so that the accesses will occur in parallel, rather than serially. Examples of this would be source code sould be on a seperate drive than your temp files which should be on a seperate drive from libraries.... Mind you that these need to be seperate drives and not seperate partitions on the same drive which won't gain any performance.

Since you are concerned with reliability -- Get at least one large drive that is used for backups of the other drives. Then regularly backup! Let me repeat that because it is important -- regularly backup!

I apprciate your advice, but the physical hardware upgrades you mentioned are not possible both from physical limitations (motherboards supports 2GB of RAM max) and cost (A64 X2s). I couls swing a 3rd HDD to help spread the load around and just might try that.

 

[LiamC]

If cost is the limiting factor, your HDD arrangement is about as good as it gets. You will want each hard drive on a differrent channel--easy to do with SATA, harder with IDE/PATA.

I'd suggest 1 PATA as your boot
2 x SATA.

partitioned as appropriate.

Get yourself an Athlon X2 3800+

I do development in Delphi, Java/J2EE using WebSphere & COBOL. I only got the X2 yesterday so these a rough.

Delphi-186 000 lines of source, 40 odd unit files, Full build--a few seconds(~6) compared to a S754 3200+ with same disc arrangement that would take about twice/three times that.

WebpShere--120+ class files--night and day. Debugging is so much faster it isn't funny

COBOL, 120+ source files. IBM VisualAge, generates 3 files + OBJ for each source. 20-25 seconds versus more than a minute, could be two on s754.

 

[jps0]

If cost is the limiting factor, your HDD arrangement is about as good as it gets. You will want each hard drive on a differrent channel--easy to do with SATA, harder with IDE/PATA.

I'd suggest 1 PATA as your boot
2 x SATA.

partitioned as appropriate.

Get yourself an Athlon X2 3800+

Believe me, I would love to get one!!!

There is one problem, I am not paid for *any* of my coding work and I do not have $625.00 minimum to spring for the new CPU, motherboard, and PSu that would be required. My work is all volunteer for the Mozilla group. I code as a hobby on the side and the donations I do get all go towards covering the cost of webhosting fees for the builds I do release.

 

[LiamC]

At least you have an Athlon. At work, I develop on a 1.6GHz Willamette P4.

The Java compiles I listed take many minutes at work...

 

[jps0]

At least you have an Athlon. At work, I develop on a 1.6GHz Willamette P4.

The Java compiles I listed take many minutes at work...

I am just trying to maximize what I *do* have!

 

[LiamC]

At least you have an Athlon. At work, I develop on a 1.6GHz Willamette P4.

The Java compiles I listed take many minutes at work...

 

[LiamC]

D'OH!

 

[tazwegion]

How about 4 x 80Gb SATA (quite cheap @ the moment) drives in a RAID 0 + 1 configuration... speed of disk access & mirroring for security of data? some additional RAM would be nice also

 

[jps0]

How about 4 x 80Gb SATA (quite cheap @ the moment) drives in a RAID 0 + 1 configuration... speed of disk access & mirroring for security of data? some additional RAM would be nice also

Fine, how do I distribute source, libraries, output, dev environment, and OS across that?

 

[tazwegion]

Clean install, on boot-up (you will need to consult your mainboard manual) configure the included SATA controller driver/software (included in bundle) for the for drives nominating those in RAID mode 0 & 1 accordingly... prior to installing OS, sorry I can't be more specific than that but I use my SATA's in a single non-RAID configuration, there are others about this forum however (Merc' for example) who build RAID storage systems regularly

Alternatively here is a Beginners guide to RAID arrays @ PCstats, which covers both software & hardware RAID configurations

 

[jps0]

Clean install, on boot-up (you will need to consult your mainboard manual) configure the included SATA controller driver/software (included in bundle) for the for drives nominating those in RAID mode 0 & 1 accordingly... prior to installing OS, sorry I can't be more specific than that but I use my SATA's in a single non-RAID configuration, there are others about this forum however (Merc' for example) who build RAID storage systems regularly

Alternatively here is a Beginners guide to RAID arrays @ PCstats, which covers both software & hardware RAID configurations

I know how to make arrays - that is not my question. What i want to know is where to place the different components of a development environment, i.e. source, libraries, bins, etc, for maximum throughput!

 

[Tannin]

Last time I ran an IDE in any serious way it was Turbo Pascal under DR-DOS 6.0. (Possibly it was some other DOS version - but you get the idea as to general era.) Nevertheless, I think I still have sufficient grasp of the basics to comment usefully.

The bottom line, I think, is that provided you avoid any of the especially unsuitable setups (i.e., so long as you don't do anything totally stupid, which you won't), you won't get any genuine, noticable, real-world gain by tweaking the configuration.

I better explain what I mean. You are already planning to use however many drives you have sensibly (i.e., split the workload up between the different units). Any of the several methods discussed above will do that quite effectively. Let's pluck a figure out of the air and say something like 90% effective (as opposed to the most perfect possible split on your current hardware, which would = 10%). From here on in your gains are going to be marginal. My guess is that they will be very marginal. The days when carefull tuning of application usage of consumer-grade hard drives was capable of producing naked-eye-noticable real-world gains are long over. When we had 70ms seek times and 1MB RAM, this stuff was important. You can still f*-up a system by setting it up very badly these days, but it's pretty hard to improve it much beyond doing the fundamentals - which you have already done.

So what can you do?

Multiple single drives and RAID solutions can only go so far. RAID is probably a particularly bad way to go, as RAID is essentially all about improving transfer rates, where your problem is almost certainly all about access times and caching. Higher transfer rates are unlikey to make any noticable difference at all. You've already got high transfer rates. Even the cheapest, crappiest drive made today has a very goot DTR.

What about caching? You've already got (almost) all the caching it is practical to have. Sure, Windows is crap, we all know that, but the default Windows hard drive caching system is remarkably effective. You can get a good handle on just how effective it is by considering how many third-party HDD caching software packages are current on the martket today: none. Compare to 15 years ago when there were several, and at least two of them (probably more than two) provided instantly visible improvements. (Norton Cache and Super PC-Kwik - Norton cache was quite fast and quite buggy: Super-PC-Kwik was very fast and to the best of my knowledge completely bug-free.) No-one bothers with this sort of add-on anymore for the very good reason that you get something just as effective free when you buy Windows.

Hardware cache? Nope: if you look into the performance boost provided by caching hard drive controller cards, you'll soon discover that they are quite expensive and (except in certain specialised applications that need not concern us here) don't really achive anything that Windows (of Linux, or whatever you use) cannot do just as well, provided only that you have enough main system RAM. It's much cheaper just to plug more main RAM into your motherboard, and more effective — because the main system RAM can be used for many different purposes, more system RAM can be pressed into service wherever Windows thinks it will be most useful, not just for a hard drive cache.

 

[jps0]

Tannin
Thank you very much for the explanation - it does make sense...

 

[Tannin]

That leaves access times. This is where the bulk of your disc-related performance problems will lie. There is only one way to get a significant improvement in access times: spend the money. Either you buy a Seagate X15 (or one of the various similar drives from other manufacturers) or you go without. It really is that simple. You can buggerise about with a million other tweaks and twists and combinations, but the bottom line is that none of the consumer-level drives from any of the manufacturers is a great deal faster than any of the others. (Western Digital's 10K IDE drives may be an exception, but I wouldn't recommend using one of those to store anything that actually mattered to me. Ask David During what he thinks of all those Rators he bought a while back - he's a regular here, and will probaly pop into this thread before too long.)

With an X15 (or similar) you will get a massive improvement in your access times, and (if your task is storage-limited) a corresponding massive improvement in your system responsiveness. Without an X15 (or something similar), you'll get small change at best.

What about other possibilities? So far, I've noticed three main suggestions in this thread: more RAM (P5-133XL), faster CPU (also suggested by P5), and a faster hard drive (my suggestion). All three are expensive. All three are likely to do some good. Which of the three is likely to do the most good? I don't know. My guess (and it's just a guess) is HDD, CPU, RAM in that order, but this will depend entirely on your application mix, and that I'm not going to speculate about - it's just been too long since I ran that kind of stuff. If you pointed a gun at my head and said "I want your best guess right now!" I'd go for the X15 .... but I could easily be wrong.

All in all, this is probably not what you want to be reading, JPS. It would be much more pleasant to read a post that says "just do X and you will get Y% improvement". But there you have it: to the best of my knowledge, there is nothing you can do (short of spending lots of money) that is likely to produce a dramatic enough improvement to be worth the trouble of doing it.

That is a fantastic project you are working on, and I really admire the people who have made it what it is today, so if there was a way to help you do still more of it and better, I'd love to suggest it. But unless someone here comes along with a bright idea I haven't thought of yet, I don't think there is anything much that I can suggest.

 

[jps0]

Tannin
Of all the places I have inquired around, you have been the most helpful and I do thank you. Right now, I am maxed on RAM (2GB is the limit of my motherboard). Upping to an A64 3800+ X2 would be a great move, but spendy as it would mean at least another $625 for mobo, CPU, and PSU. Now the X15 is attaractive, but which one - they all look to be 18.4GB but have different pin arrays and different prices? I just searched Newegg and they don't even stock 'em. Can you recommend a vendor?

Thanks again...

 

[tazwegion]

Sorry, I misunderstood the question (I go off on a different tangent sometimes :roll, I assumed your issues were with speed of access (to data) & security (of data), I'll disagree with Tannin on this one and feel confident that even if one of the primary striped drives failed, due to the second RAID 1 (mirrored array) you'd still have a working RAID 0 with access to your data.

The idea of separating all your components works for me also... though backing up said media from 4+ drives to a much larger 250GB+ drive would be prudent

As modern platforms support up to 4 SATA's & 2-4 IDE's you could use a mix as already suggested by LiamC... perhaps recycling/sourcing an IDE drive you currently own for the primary boot disk, with the money you save being funneled back into extra RAM or faster CPU

 

[blakerwry]

Those are some great posts Tannin. Very insightful and well thoughtout.


JPS0, I also admire what you're doing. If I had any spare parts that would help you I'd be happy to send them your way, but I'm afraid all I have are leftovers from several generations back.

 

[jps0]

blakerwry
I do appreciate your comments!

ALL
For the curious, here are my contributions: http://moox.ws

 

[blakerwry]

I've used your builds for the last year or so at work, nice work. Welcome to the forum!

 

[Tannin]

Thankyou gentlemen! I feel flattered. Not sure about helpful though: all I really said was "don't do anything except maybe spend quite a lot". Perhaps saying what to not bother doing is useful, in a negative sort of way.

I don't think more RAM is a good idea. You've only just bought a new mainboard and CPU, so swapping them over for something that takes more RAM so soon is a bad economy. Wastes most of the money you have spent so far. Much the same applies to CPU.


I guess that leaves us with three questions to answer, JPS.

* 1: Will an X15 (or etc) provide a worthwhile boost?
* 2: If yes, can you afford one?
* 3: If yes, which one, where do you buy it?

I'll address them in reverse order. I'm not a good person to answer the second part of #3 as I'm based In Australia, so my suppliers won't help unless you are also down under. As to the first part, there are (if I remember correctly) two current sizes, and a host of connection options. The 36GB unit is half the price of the 72GB unit. If 36GB is enough to put your speed-critical stuff on (I imagine this would be source code, includes, and temp files), then it's a possibility. Just the same, it wouldn't leave you a lot of room to grow.

PS: Hello, I see Seagate list a 146GB X15 now. It doesn't appear on my wholesale pricelists, so it must be a new model. No matter, the price will make you blanch, if I'm any guess.

I haven't kept current with the connection options either, alas. The best answer always used to be standard LVD SCSI, and it probably still is, but there are a whole stack of new-fangled ways to connect fast hard drives that I've not kept up to date with: SAS for one, IDE to SCSI converters for another. It may already be practical to attach a SCSI drive like an X15 to a system without buying an expensive SCSI controller card. Someone here will know.

My own X15s are quite old now. I have two 1st generation ones, and a Mark II; all 18GB. Two of them are doing just fine, space-wise, the third one is starting to get a little tight and I'll have to reorganise a bit shortly. Desite their age, they are light-years faster than any other drives I own, or have seen come into the workshop.

Cost here in Australia for a 36GB X15 is about AU$380 plus tax. Double that for the 72GB models. My old 18GB units were about AU$900 each. The previous batch of fast drives I had before that (a 1st-generation 4.5GB 10K Cheetah Mark 1 and a 9GB IBM Ultrastar equivalent) were close to AU$2000 each — so although good SCI drives are still expensive, they are a lot cheaper than they used to be.

Question 2: can you afford one? That probably depends on how much you want one, and that in turn on how much difference it might make. So:

Question 1: will it be worthwhile? We need someone who knows a lot about the performance bottlenecks of your software setup to answer this. Where is the bottleneck? Is the system waiting for data most of the time? Or is it busy crunching (i.e., CPU bound)? If it is data off the drives that holds it up (quite likely), then an X15 (or etc) is likely to be useful. It is especially likely to be useful if you are accesing lots of small files, or accessing lots of small segments from larger files.

Let's imagine two scenarios:

Scenario 1: your compiler reads in a series of source-code files, then a list of include files, stores temp results in a stack of temp files, writes output to a series of executables. Lots of semi-random disc accesses: nothing does this as well as an X15. With 3.5ms seek time and low latency as well, you are getting to the data roughly 3 times faster than with a standard 7200 RPM IDE drive.

Scenario 1a: Same as (1), but the included data isn't in individual files, it's scattered around in a small number of large files. (Database work is a typical example of this sort of acces pattern.) For our purposes, this is the same thing as (1) - the read heads still have to skip around all over the platters looking for your data.

Scenario 2: Most of the data your system uses is contained in a small number of large files that are read into memory in one big lump. The compiler then extracts the bits it wants from RAM. Now the bottleneck is RAM size, and to a lesser extent, data transfer rate. (Whether it's read-in time or swap-out-to-disc-to-make-room time, we don't care.) Here, a bunch of IDE drives or a RAID setup would be nearly as good as an X15, and a whole lot cheaper.

Which scenario applies to you? A simple, brute-force way to find out might be to set your system up such that your data drive and your swap file drive are seperate. Run a complie, and see which drive LED stays on all the time. If it's the data drive, you probably want an X15. If it's the swap file drive, you need more RAM.

 

[blakerwry]

36BG 15k.4
http://www.zipzoomfly.com/jsp/ProductDetail.jsp?ProductCode=101514

73GB 10k.7
http://www.zipzoomfly.com/jsp/ProductDetail.jsp?ProductCode=101577

68 pin SCSI cable
http://www.zipzoomfly.com/jsp/ProductDetail.jsp?ProductCode=315011

LSI u320 controller
http://www.newegg.com/Product/Product.asp?Item=N82E16816118034


What's odd is that newegg used to carry the u160 version for ~$50... which seemed incredibly cheap however that is no more...
Seems it is still available at other retailers though...
http://froogle.google.com/froogle?q=LSIU160&num=20&hl=en&lr=lang_en&safe=off&sa=N&tab=wf


SCSI isn't all that cheap...

 

[jps0]

Yikes! :evil:

What about something like this:
http://www.cmishop.com/store/ShowDe...MfgPartNo=&nCategorySubID=125&txtSiteTrack=FG

 

[tazwegion]

Ummmm... just curious as to which A64 2.2Ghz you meant, the San Diego core with 1Mb of cache, or the Venice core with only 512Kb of L2?

Right now, I am maxed on RAM (2GB is the limit of my motherboard).

Additionally is the 'mainboard' you mention that which you already own OR intend on purchasing?

Gigabyte GA-K8NF-9
MSI RX480M2
DFI NF4ULTRA-D
Abit AV8

All support -> 4Gb of RAM

BTW checked out that link jps0 nice work you're doing there, Mozilla FireFox is my favourite browser! :jumpin:

 

[blakerwry]

Tannin is old in his ways (you can tell because of the Moss on the exposed northern areas). The x15 is an older generation drive. What you'd be looking for is the 15k.3 or 15k.4 model from Seagate.

Fujitsu or Maxtor drives would also be good and comparible in cost and speed.

 

[blakerwry]

Looks like HyperMicro carries the slightly older 15k.3 unit in both new and recertified condition for considerably less than ZZF

http://hypermicro.com/search_detail.asp?dept_id=06-005&sp=15k.3&sp_flag=0

 

[Tannin]

I have one of those, JPS. (Or is mine the previous, slightly slower, model? Previous model, I think.) Either way, it goes like the proverbial substance off a shovel. I'll have it in service for quite a few years yet. Old, but good.

But what is it doing in stock as a "new" drive? It should be OK, but a drive that old still listed as a new product, and with only a 90-day warranty instead of the standard 5 years .... ???

Maybe someone here has a spare one.

 

[Tannin]

Tannin is old in his ways (you can tell because of the Moss on the exposed northern areas). The x15 is an older generation drive. What you'd be looking for is the 15k.3 or 15k.4 model from Seagate.

They are current model X15s, and the drives I was talking about. It's just that some fool in the marketing department switched labels on me. (Why do the idiots have to change names all the time? Nvidia have done the same thing with their video cards. You show somebody an FX5700 or a 6600GT and they say "what the hell is that?". Describe them as (respectively) a Gforce 5 and a Geforce 6 and they say, "oh, fine, I'll buy one".)

And yes, I always like the Seagates myself, but Hitachi and Maxtor will have similar products. Probably doesn't much matter which of the three brands you buy.

 

[jps0]

Ummmm... just curious as to which A64 2.2Ghz you meant, the San Diego core with 1Mb of cache, or the Venice core with only 512Kb of L2?

Right now, I am maxed on RAM (2GB is the limit of my motherboard).

Additionally is the 'mainboard' you mention that which you already own OR intend on purchasing?

Gigabyte GA-K8NF-9
MSI RX480M2
DFI NF4ULTRA-D
Abit AV8

All support -> 4Gb of RAM

BTW checked out that link jps0 nice work you're doing there, Mozilla FireFox is my favourite browser! :jumpin:

Thanks!

As for the mobo and CPU combo, I have an A64 2800+ that is overclocked to 2.2GHz on a Soltek SL-K8AN2E-GR motherboard. It is a first-gen Socket 754 system - I just could not wait to try A64!!! At the time I bought it was the best I could afford. I know the newer socket-939 system will do 4GB for RAM, but I am not going there until I upgrade my CPU as well.

 

[blakerwry]

While I hopefully made it easier to find what you would want in SCSI gear. I attempted to make it easier to see the expense as well.

I don't personally think you will get a significant (or even noticeable) increase in performance by going SCSI over a 2 or 3 disk independent ATA solution.

You'll probably be best off waiting for the money to upgrade your CPU and motherboard.

 

[tazwegion]

As for the mobo and CPU combo, I have an A64 2800+ that is overclocked to 2.2GHz on a Soltek SL-K8AN2E-GR motherboard. It is a first-gen Socket 754 system - I just could not wait to try A64!!! At the time I bought it was the best I could afford. I know the newer socket-939 system will do 4GB for RAM, but I am not going there until I upgrade my CPU as well.

Ahh! that explains the 2Gb maximum RAM support

Well considering that factor... drives is the ONLY logical upgrade choice you have, unless you'd want to purchase an A64 3700+ (s754) which has 1Mb of L2... though that could be considered a waste with your proposed future upgrade to s939 anyways

SCSI... Tannin has been praising that media since I first visited his PC store in '97 :lol:

 

[Bozo]

I presume you are running a 64bit operating system, right?

XP 64bit apparently uses the same hard drive controller system as 2003 Server. In my limited testing, XP 64bit is faster than standard XP.
I believe MS had a trade up program that would allow you to trade 32bit XP for 64bit XP.
But then there is the driver issue. And, will your programs run on XP 64bit.
If you are running XP 64bit, open Device Manager, select your hard drive, open the policies tab and make sure all the performance boxes are checked. They are not by default.

Bozo :mrgrn:

 

[LiamC]

jps0, my thanks for your MOOX builds as well, I've been using your M builds for a while. Unfortunately there are a few reprobates on this board who think Opera is the better browser! As if!

BTW, is the work crunch off yet, or do you still have a while to go?

Back on topic, are you actually using more than 1.5, 2GB of RAM. Even with a couple of instances of WebSphere going plus the usual other apps and the debugger kicked in, I struggle to use more than 1.5GB