Folding@Home

[LiamC]

I only had one or two A5's in the last week or two. All A3's. And it's getting hot...

 

[Bozo]

I tried the Beta 7 Client Control program for the last few days. Output was down compared to version 6.xx.
Going back to the 6.xx version.

 

[CougTek]

This is insane. One of the teams behind us (Megware) has a user (fswork) that completed a single unit worth 616248 points. In order to do that, according to the bonus point calculator, you have to complete a big unit with a "time per frame" of 18 seconds and don't take more than one minute to upload the result. No consumer-grade computer can do that. Even a four-socket board with four ten-core CPUs couldn't do that. IMO, someone is using a university super-computer to boost its stats. It's the only way. It's also unfair.

 

[Handruin]

They produced an even larger single unit of 629,369 on 10/30. That really is a ridiculous system. Looks like they've been sustaining it for quite some time.

 

[CougTek]

An unoverclocked i7 2600 has a TPF of ~31 minutes for a big unit (other than the 2684). An i7 2600 is a 4 cores/8 thread 3.4GHz CPU. So assuming the FAH client scales perfectly, then what's needed to shave the TPF from 31 minutes to 18 seconds is roughly 103 Core i7 2600 CPUs. Or 412 cores/824 threads at 3.4GHz on a SandyBridge architecture.

I haven't seen blade servers with more than two sockets for Xeon E7 processors. So let's assume that we can fit two Xeon E7 8870 per blade. That's 20 cores/40 threads of Sandybridge marvel per blade, although only at a 2.4GHz frequency. You can fit up to eight of these in a full-height blade enclosure and a blade enclosure is 10U size. So in a 10U size, you'll be able to fit 320 Sandybridge cores at 2.4GHz. 3.4GHz (i7 2600) is ~1.417x 2.4GHz, so if the FAH client scales perfectly, you'll need ~584 cores at 2.4GHz to match 412 cores at 3.4GHz. At 20 cores per blade, you'll need at least 30 blades server, or almost four full blade enclosures, to achieve a TPF of 18 seconds.

At the electricity cost in Massachusetts, how much would it cost to power up 30 blade servers, with two 130W CPU each, for ~31 minutes? Because that's how much that sucker should give to his University/big ass employer every time he computes a big unit on their equipment.

 

[Bozo]

Maybe it's someone at a Google or Facebook data center?

 

[CougTek]

So in a 10U size, you'll be able to fit 320 Sandybridge cores at 2.4GHz.

That should have been 160 instead of 320. I forgot to correct it when reviewing my post. The 30 blade servers is still good however.

 

[CougTek]

Maybe it's someone at a Google or Facebook data center?

He could, but he doesn't need to. There are several other companies that own mainframes or large amount of blade servers. There are other ways to do it than blade servers, but it's probably the highest density way. There are 4 sockets 1U designs, so in theory you could fit 4x 10 cores = 400 cores in a 10U space, but I'm pretty sure a 10U sandwitch of 520W (4x130W) systems (probably more than 600W once you calculate power loss and other components) would severely overheat. In any case, no one has that kind of processing power in its basement. That's large-company or university territory. I'm almost certain even SSDdrueding's company doesn't have that kind of processing power. Handruin's company does, though.

 

[Handruin]

In my little area of the lab, we're occupying one floor tile for compute and inside it we have:

1 x 10U HP c7000 chassis
15 HP blades
10 = BL460c G1
2x Intel E5450 @ 3GHz (quad core)
5 = BL460c G6
2x Intel X5550 @ 2.67GHz (quad core)

3x 2U Dell R710
2x Intel X5680 @ 3.33 GHz (hex core)

The HP blade center (which is running the 15 blades) has a 24 hour power average of 3723 Watts and the power usage is sampled every 5 minutes. I don't know what my company pays for electricity, but at home I think it's about $0.156/kilowatt hour (I need to double check that). Also keeping in mind that the blade center is not running at 100% CPU usage. The chassis is also powering four Cicso network switchs and two brocade fibrechannel switches (and also a boatload of fans). I'm certain this average would go up if all CPUs were maxed out due to F@H.

I don't have an accurate recording for the Dell systems.


Coug, the user who is crunching a single unit is doing so on a single massive platform. I don't think it would be possible to share the workload of a full blade server unless F@H has some kind of clustered client?

 

[CougTek]

Then it has to be something designed by Cray or similar.

 

[mubs]

I know nothing of all this, but will still risk shooting my mouth off: he's not using GPUs to achieve these fantastic results?

 

[BingBangBop]

I think you made an error with your choice of WU's. I used a 6904 and the time per frame was aprox. 13 minutes (not 20 seconds) to get 800K. A time per frame of 80 minutes still gave a 300K points. A TPF of 81 minutes only gave 31K points (exceeded the preferred deadline?).

I would contend that he got something like one of those and had a really fast machines (aprox 6x faster than the minimum necessary to get any bonus points). If a 2600K can complete in 80 minutes, then perhaps a quad socket six core equivalent Xeon could finish it 6x faster and then there is also the possibility of an 8-socket machine. I just started thinking of the recent $75-$100K Santilli dream machine build and how well it would fold.

Also, when I looked at the stats, it was obvious that the machine was not dedicated to folding for it was not producing one of these every day

 

[CougTek]

You can't crunch a big unit with the GPU client and I don't see how you can shave the TPF of a regular GPU unit to the point of having over 600K points for it. It has to be a big unit.

 

[CougTek]

I wasn't awared that units as big as the Project 6904 existed. It must need a massive amount of processing power to complete it within the prefered deadline.

 

[BingBangBop]

I would assume that the project isn't offered to just anyone but only to those with enough processing power to complete it in time. Normal folk need not apply and would never see such.

 

[CougTek]

I read that Grandpa (from Asus Republic of Gamers) received project 6903 and 6904 units. The guy only has i7 970 and 980 systems. He average 340K ppd. There's something we do wrong. My two i7 2600, one overclocked to 4.3GHz, are almost as powerful as his setup and yet I barely average 50K ppd. SSDdrueding could in theory match Grandpa's output because he has similar computers.

Do you imagine what would be our global production if we would have two or three members with these numbers?

 

[Bozo]

Would having multiple video cards produce that kind of results? Some motherboards can handle up to 4.

 

[Bozo]

Megware specializes in Linux clusters. Maybe they were stress testing a new setup?

 

[ddrueding]

Would having multiple video cards produce that kind of results? Some motherboards can handle up to 4.

But each video card crunches its own unit...they don't team up on a single unit.

 

[CougTek]

Megware specializes in Linux clusters. Maybe they were stress testing a new setup?

Ah, it could be an explanation. Then maybe that user isn't an ass hole.

I will investigate how to get a mega big unit and what hard ware it takes to finish it on time.

 

[ddrueding]

When you find out let me know...I occasionally get to play with decently sized toys.

 

[Handruin]

I'd like to know also. Once Ivy Bridge gets here next year, I'll want to spec one to run these big units if it's at all possible.

 

[CougTek]

I've found it.

So, in summary, you need a 12 cores system running 64bits Linux and you need to add the -bigbeta flag. On an i7 9x0 systems, you need to massively overclock the rig in order to complete the WU in time. Some user complained that his 4.0GHz i7 980X would miss the prefered deadline by a few hours, but Grandpa completes them with his i7 980 at 4.35GHz.

You won't get them on Windows.

It's pretty step requirements, but for a unit giving at least 217K points (projet 6903) or 335K points (project 6904), it's worth it.

 

[ddrueding]

12 cores including hyperthreading, so a single 6-core system will do it if clocked high enough, and a dual 6-core system could do it without significant overclock? Would be interesting to throw on some of the bigger VM servers while the VMs are down for maintenance, or on a backup/redundant machine. After hours, vMotion everything onto one piece of hardware and load up a dedicated Linux VM for some crunching?

 

[CougTek]

You won't make the prefered deadline on a 2x 6 cores/12 threads system if you only crunch the unit in after-hours time. A 4.2GHz i7 9x0 needs five continuous days to complete the unit in time for the bonus to apply. Don't forget it has to run on Linux 64 bits.

 

[CougTek]

I need to sell all my gears in order to get a decent LGA2011 system to run this. LGA2011 processors are out on Friday. We should see motherboards next week.

 

[CougTek]

...or maybe not. SandyBridge-E doesn't appear to be much faster than the 6-cores i7 9x0. It might make more sense to just get two 6-cores i7 980 for the two spare x58 motherboards I have.

 

[CougTek]

I RECEIVED A PROJECT 6900 UNIT! WOOHOO!!! I RECEIVED A PROJECT 6900 UNIT! WOOHOOO!! WOOHOOHOOO!!!

It was about damn time.

 

[Handruin]

Sweet. Can't wait to see how it works out for you.

 

[timwhit]

Coug, I think you have an addiction.

 

[CougTek]

Yes I am sadly adddicted to FAH. It's a costly addiction too, but probably less than cocain. I think I'm better than Mark in that regard because he never admitted to be addicted to FAH and his effort has been even more insane than mine so far. If I succeed to replace my SandyBridge by a 6 cores i7 980, then it might change (I'll be even sicker than Mark).

 

[CougTek]

Just in case David goes crazy : EVGA dual socket 1366 board at 450$ instead of 600$.

 

[ddrueding]

Nah, I can wait for the next generation before doing anything else.

 

[CougTek]

Just in case... Remember that this is without the -bigbeta flag. It would be much higher with it.

 

[ddrueding]

They're only running GTX460s...lame

 

[CougTek]

The review has been done in early 2010.

 

[mubs]

Just in case David goes crazy : EVGA dual socket 1366 board at 450$ instead of 600$.

I thought EVGA was to be avoided like the plague. Has that changed so much that they've become a tier 1 player now?

 

[P5-133XL]

I'm sure some people here won't like it, but it is the way it will be!

NOVEMBER 14, 2011
Planned changes to "Big Advanced" (BA) projects, effective January 16, 2012

Big Advanced (BA) is an experimental type of Folding@home WUs intended for the most powerful machines in FAH. However, as time goes on, technology advances, and the characteristics associated with the most powerful machines changes. Due to these advances in hardware capabilities, we will need to periodically change the BA minimum requirements. Thus, we are shortening the deadlines of the BA projects. As a result, assignments will have a 16 core minimum. To give donors some advance warning, we are announcing this now, but the change will take place in 2 months: no earlier than on Monday January 16, 2012.

We understand that any changes to how FAH works is a disruption for donors, and we have been trying to minimize such changes. For that reason, we are not changing the points system at this time.

However, we want to emphasize that the BA program is experimental and that donors should expect changes in the future, potentially without a lot of notice (although we will try our best to give as much notice as we can). In particular, as hardware evolves, it is expected that we will need to change the nature of the BA WUs again in the future."

Announcement
Discussion

 

[CougTek]

Well, it's a good thing I haven't spent almost 600$ for a Core i7 980 yet. What a waste it would have been. Since the price of 16-core systems is prohibitively high, it means it's the end of the road for big advanced units for me and almost everyone else here as well.

It also means I no longer have any mean to take the team #1 spot from Mark.

 

[ddrueding]

16 Cores? Hmmm.