New motherboard supports 288GB of RAM

CougTek

Hairy Aussie
Joined
Jan 21, 2002
Messages
8,729
Location
Québec, Québec
It's sure to be popular with the West Coast cartel (SSDdrueding and Santilli) :
http://www.gigabyte.sg/products/product-page.aspx?pid=3508

3244.jpg


That's a VM monster machine. Only 470$, supposedly.
 

MaxBurn

Storage Is My Life
Joined
Jan 20, 2004
Messages
3,245
Location
SC
12 CPU and 288 GB of memory on one board. We have come a long way.
 

ddrueding

Fixture
Joined
Feb 4, 2002
Messages
19,747
Location
Horsens, Denmark
That is a nice looking board. Not on Newegg yet, and I would need to wait for another HPC project to justify one. None of my clients have enough VM load to even use half that.
 

BingBangBop

Storage is cool
Joined
Nov 15, 2009
Messages
667
It's not the MB that's going to cost that arm and leg rather it the only supported 16GB RAM DIMM. Don't forget the price of two pricey Xeon's too.

Try $1305.28 +S/H or $23,500 for 18
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
It's not the MB that's going to cost that arm and leg rather it the only supported 16GB RAM DIMM. Don't forget the price of two pricey Xeon's too.

Try $1305.28 +S/H or $23,500 for 18

I get that it's expensive, but the specs on the website claim it supports 8 or 16GB DIMMs.
 

BingBangBop

Storage is cool
Joined
Nov 15, 2009
Messages
667
I get that it's expensive, but the specs on the website claim it supports 8 or 16GB DIMMs.

Not to get to 288GB and that is the point of this MB. If you use 8GB DIMMs, it only supports 96GB total (not even 144GB or half).
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
Not to get to 288GB and that is the point of this MB. If you use 8GB DIMMs, it only supports 96GB total (not even 144GB or half).

Sure, I'm able to do the basic math of 18 x RAM module size. I was commenting on you saying it only supported 16GB RAM modules.
 

BingBangBop

Storage is cool
Joined
Nov 15, 2009
Messages
667
I think what B3 meant was that, to achieve 288GB, there is only one approved 8GB DIMM.

No, only the 16GB DIMM will get to 288GB and it costs well over 1K each. It is not possible for the 8GB DIMM to get there or even close.
 

BingBangBop

Storage is cool
Joined
Nov 15, 2009
Messages
667
While not technically supported, I'm sure that if you don't care to reach large RAM sizes it will run happily on standard 1GB DIMMS (probably not on the entire 18 DIMM capacity though). But then there will be lots of alternative MB's and what is the point of explicitly buying this MB. The only real purpose of getting this MB over others is the RAM capacity which means using the 16GB DIMM.
 

Santilli

Hairy Aussie
Joined
Jan 27, 2002
Messages
5,286
I bought dual Xeons because at the time, they, and the scsi enabled motherboard seemed like a good value. In retrospect, thanks to Adaptec's pathetic zero raid card, I ended up not even using the onboard scsi. Still, the duals were fast, and reasonably priced at the time, compared to other processors, and ram wasn't bad. The little girl is still going.

That said, I haven't priced the processors, but, the ram for this board is absurd. 1200 bucks for 16 GB?

It does bring up a bigger question: What is going to happen when SSD storage space, and bus speed, are so fast that RAM becomes of questionable benefit?

Would it be faster to just use a REVO drive, on a cheaper motherboard, and the normal type of ram? Is there really much difference in speed these days between a Revo drive and RAM access?

Seems to me David's and my ram disk tests ran about 4 GB a sec through the bus on the current gigabyte boards.

The REVO's are pushing 1 Gig a second on the new generation, aren't they?

Is anyone going to be able to tell the difference?

How about a second REVO drive as a page file drive?

GS
 

Santilli

Hairy Aussie
Joined
Jan 27, 2002
Messages
5,286
I was going to say nasty things about this board, but, it's growing on me.
Your first cpu would have 6 processors, a Passmark score pretty much as fast as anything on the planet, for a discounted 1600-1700 dollars.
For 300 dollars one might decide the -i7 2600 with a PM score at 85% of that might be a better value. Or better, the i7 2600K which has gone up to 430. but, is 10/11 as fast as the 5690.

Kind of curious: Do the current Xeons do something magical that justifies price tages well over a thousand dollars, for performance slower then my 940?

Then another 16 GB ram stick, and, another 1200 dollars. So, you are in for 3300-3500, with room for expansion. Everytime you have 1200 laying around....

Odd part of my experience was that when I assessed the price value point on buying the Xeons, the value price point stayed there. The slightly faster processors never came down much in price, and never justified buying new processors for that board.
Also, ram never came down much either.

So, I wonder if that is going to be the case with this setup?

Is so, I think I'll pass on buying in at something that gives 15% improvements in performance, for 4-5 times the cost.

Still never got a real answer. Despite latency, can anyone really tell the difference in ns,
that is as a human being?
 

ddrueding

Fixture
Joined
Feb 4, 2002
Messages
19,747
Location
Horsens, Denmark
The "System Management" ones allow you to control the machine remotely, even in the BIOS. The rest are RAID cards or RAID expanders.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
Still never got a real answer. Despite latency, can anyone really tell the difference in ns,
that is as a human being?

Yes, during any kind of high CPU computationally workload where the CPU is moving memory in and out of different locations, adding 10,000 times longer latency will translate into a dramatically slower end result. There would also be greater timing and synching issues, especially in multi-socket motherboards which you seem to love. I believe to help address those issue (and other issues), the RAM is typically located as close as possible to the CPU and also with equal length paths (if possible). If you remove RAM from the picture and communicate with storage via your SATA bus...you're asking for trouble.
 

Santilli

Hairy Aussie
Joined
Jan 27, 2002
Messages
5,286
At 1200 bucks for 16 GB of ram, you BETTER be able to overclock it;-)

I can see it now: everytime I come up with another 1200 dollars, another 16 GB of ram. Sort of like the 64.5 IMSA Mustang funny car I built to be street legal...
A SERIOUS MONEY PIT. At least I'd get a tax break on this one...
 

Chewy509

Wotty wot wot.
Joined
Nov 8, 2006
Messages
3,359
Location
Gold Coast Hinterland, Australia
Kind of curious: Do the current Xeons do something magical that justifies price tages well over a thousand dollars, for performance slower then my 940?
Not anything that matters to a normal consumer whose always lived with single socket systems. What the extra $$$ gets you:

1. Dual/quad socket installations.
2. Reg ECC RAM Support.
3. Additional validation at the factory (so they are less likely to develop faults).

And in some instances:
4. More L2/L3 cache.
5. More cores (Xeon E7s now come in 10core flavours).
5. All VT modes/extensions (VT-x, VT-d, etc).
6. Higher Turbo Frequency Scaling**.
7. Quad channel RAM controllers. (All i7's are either dual or triple channel).

For single socket Xeon's they tend to be only about $20-$30 more than the equivalent i7 here, it's the dual/quad enabled ones that cost you more. eg, the Intel® Xeon® Processor E7-8870 (30M Cache, 2.40 GHz, 6.40 GT/s Intel® QPI, 10core) has a list price of just $4616.00 each. :mrgrn:

**. What this means is that the turbo frequency may kick in sooner than on an i7, eg Max freq will be with 2 cores at 100%, and not 1 core at 100%. However rumour has it, this is because Xeon's actually come 1 frequency level below what they are validated for. Eg if the chip was going to be a 2.6GHz chip, it ships as a 2.4GHz chip, hence have more room for clocking upwards. This downclocking, also aids in stability when all cores are maxed as well.
 

Santilli

Hairy Aussie
Joined
Jan 27, 2002
Messages
5,286
Thanks, that was a great post. Right now, I'm having a hard time imagining anything that an i7 940 can't do. I have barely got anywhere near the potential of this processor. I can't imagine a dual Xeon setup, with 4-5 times the processor speed, and absurd ram speed, and, add revo drives...

What on God's earth do you need that much processor for?

I gather MSFT Exchange??;-)
 

BingBangBop

Storage is cool
Joined
Nov 15, 2009
Messages
667
Come on, why limit your mind -- Go for an 8-way Xeon setup: SuperMicro SuperServer 5086B-TRF with 64 DIMM slots each capable of 16GB or a total capacity of 1TB of RAM. If you don't want it prepackaged by Supermicro, you can buy the MB and expansion cards separately and roll your own machine. Be prepared to have to mortgage your home though to pay for the beast. It wouldn't surprise me if you paid $75,000-$100,000 to get one fully loaded.

Just Do It! It will probably take 10+ years before a single CPU system will be able to compete.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,303
Location
I am omnipresent
I gather MSFT Exchange??

Exchange murders RAM and disk I/O. It's not really that rough on CPUs.

But 80 CPU cores and 2TB RAM seems to me like it would be pushing into mainframe territory. I have no idea what mainstream application that would have besides hosting virtual machines. And it would probably be cheaper and faster to get a few racks of blade servers instead.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
Exchange murders RAM and disk I/O. It's not really that rough on CPUs.

But 80 CPU cores and 2TB RAM seems to me like it would be pushing into mainframe territory. I have no idea what mainstream application that would have besides hosting virtual machines. And it would probably be cheaper and faster to get a few racks of blade servers instead.

That would be a scary number of eggs in one basket for a VM server. However, that's some nice density for a 5U chassis.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
Buy two and put them into some sort of fail-over system. :eek:

Even then, if you go with a modest 10:1 VM:core, you're talking 800 VMs on that box. Speaking from my egocentric-vmware point of view, it would take a very long time to vmotion/restart that many VMs when using HA. If fault tolerance is used, there are some pretty stringent requirements for that to work and even then it would require ample amount of network connectivity. That server only comes with 1Gb NICs, so a few 10Gb nics would be required to make this thing usable.

If I were going to get two of them and use 10U of rack space, I'd rather go with a blade server setup like Mercutio mentioned which also takes 10U of space. I could get 16 half-height HP blades in a 10U location with 2 x 6-core Xeon and 192GB RAM each with full 2x 10Gb NICs on a back plane and Cisco switches for network and Brocade 8 Gb SAN. That would work out to 192 cores and 3TB RAM for the blades vs 160 cores and 2TB RAM for the 5U server. I'd rather distribute 800+ VMs over 16 blades vs two dedicated 5U servers. It also allows for easier rolling upgrades of hardware and software.
 

BingBangBop

Storage is cool
Joined
Nov 15, 2009
Messages
667
aren't we talking about Santilli's needs i.e. browsing and light office work (lol). Blade servers are a bit overkill but an OC3 line would certainly help with that browsing.

What I would visualize as a good application for a 80 core 2TB server would be something like movie special effects video encoding rather than VM's.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
Yes, yes we are. Thanks for bringing me back on track. Santilli, let us know when you get one up and running and add a SCSI card into it. :grin:
 

Howell

Storage? I am Storage!
Joined
Feb 24, 2003
Messages
4,740
Location
Chattanooga, TN
But 80 CPU cores and 2TB RAM seems to me like it would be pushing into mainframe territory. I have no idea what mainstream application that would have besides hosting virtual machines.

Well that was a safe statement. :) 80 cores and 2T RAM is not exactly mainstream equipment either. I'm with B^3. Anything that would fall under the area of computational engineering could use this. Weather/climate modeling, structural modeling, digital efects houses, etc.
 

BingBangBop

Storage is cool
Joined
Nov 15, 2009
Messages
667
I just want to know if it supports triple crossfire ;)
From what I can tell, it will support quad crossfire with dual wide cards all running at x16 (PCI-E v. 2.0) . I looked at the manual and unfortunately, OC'ing capability seems nonexistent, which really doesn't surprise me much.

I was curious to see what kind of CPU cooling is involved. Each CPU has a heatsink, the dual CPU addon card has an air shroud and the MB has a fan header for each dual CPU card but I don't see where fan's are mounted to pump air into each shroud.
 

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
17,497
Location
USA
aren't we talking about Santilli's needs i.e. browsing and light office work (lol). Blade servers are a bit overkill but an OC3 line would certainly help with that browsing.

What I would visualize as a good application for a 80 core 2TB server would be something like movie special effects video encoding rather than VM's.

I always had the impression that Greg played a few games, downloaded tons of videos from disreputable sources, and watched them on cheap Costco TV displays. I don't think he was using heavy computing for databases or scientific modeling, etc. I could be wrong...
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,303
Location
I am omnipresent
Well that was a safe statement. :) 80 cores and 2T RAM is not exactly mainstream equipment either. I'm with B^3. Anything that would fall under the area of computational engineering could use this. Weather/climate modeling, structural modeling, digital efects houses, etc.

I would think that such things would be better off in a cluster configuration where each x CPU cores would have its own access to a pool of RAM. Even with 2TB of RAM, there's only so much bandwidth available to access memory, and anything that's going to work well in parallel across 80 cores is also going to work well with the slightly different latency model of a cluster.
 
Top