10 gig Ethernet

[Stereodude]

Anyone running 10GbE at home or played with it at work?

In my case I'm thinking to connect my backup server to my new NAS via 10GbE. Is it really as simple as buying two Mellanox Connectx-2 cards for under $20 a piece and a few meter Twinax SFP+ for under $20 and being in business?

 

[Chewy509]

We're in the process of upgrading our LAN at work to 10GbE. We're using Intel adapters, and have been doing a lot of tuning with Jumbo Frames, etc to maximise performance... (Main 10GbE LAN is between the ESXi servers, with rest of the LAN serviced by 1Gb copper).

But at the basic level, yes, it's that simple.

 

[Stereodude]

This switch could be interesting... :scratch:

Get a two SFP+ 10gig card for the NAS/server. Connect one SFP+ port to the Mikrotik switch and the SFP+ other port directly to the backup server (which has its own one SFP+ 10gig card). That would give 10gig between the two servers, 10gig from the switch to the main server, and allow another switch to be chained off the free 10gig SFP+ port on the Mikrotik for later network expansion.

 

[Howell]

I do have a redundant 10G fiber network backbone and to the server hosts. Not really applicable to the home.

 

[Stereodude]

Will a Cisco Twinax SFP+ cable work in the Mellanox SPF+ port? Does coding even matter for twinax cables or is that only for optical modules?

 

[Mercutio]

Twinax is twinax as far as I've been able to find.
Why not look in to infiniband (40Gb) if you're at the point where you're doing something as exotic as using anything that has SFP ports in the first place?

 

[Stereodude]

Twinax is twinax as far as I've been able to find.
Why not look in to infiniband (40Gb) if you're at the point where you're doing something as exotic as using anything that has SFP ports in the first place?

What's a "bargain" 40Gb InfiniBand setup cost if I just want to link two PCs together?

I'm not sure I can saturate a 10Gb link on the one side plus the one machine still runs XP x64, and I'd rather not have to putting Windows 7 on it so I'd need drivers for XP x64. Additionally, I'd expect it's not cheaper than 10GbE. So, I can pay more for no tangible benefit and give myself a headache in the process? Let me jump right on that...

Edit: Not sure I can find Windows XP x64 drivers for a Mellanox ConnectX-2 either... Grr....

 

[Stereodude]

It looks like my options are limited if I want Windows XP x64 (Server 2003 x64) support. I can't find XP x64 / Server 2003 x64 drivers for any Mellanox Connectx-2 card. The Intel x520 cards have Server 2003 x64 drivers. They're not quite as cheap as the Mellanox Connectx-2 and I don't know how well they will interoperate. I could get a Windows 7 Pro license for about the extra cost of the Intel x520 card, but then I have to redo the whole system. Then again, maybe that's the right thing to do. :scratch:

 

[Handruin]

We had a discussion about a similar topic here if you're curious:
http://www.storageforum.net/forum/showthread.php/10554-Infiniband-and-10Gb-networking

I've also spent a bit of time over on ServeTheHome.com because they have a community of people trying to get 10Gb in the home on the cheap also. I've been considering adding 10GbE to my setup but I'm unclear how the right way would be to integrate it in my own environment.

 

[Stereodude]

We had a discussion about a similar topic here if you're curious:
http://www.storageforum.net/forum/showthread.php/10554-Infiniband-and-10Gb-networking

Thanks, I didn't recall that thread.

I've also spent a bit of time over on ServeTheHome.com because they have a community of people trying to get 10Gb in the home on the cheap also. I've been considering adding 10GbE to my setup but I'm unclear how the right way would be to integrate it in my own environment.

I was doing some reading over there too. At this point I'd be good just connecting my two servers with 10GbE. I'd probably put a dual port card in the new NAS/server so I could also connect it to a switch later like the Mikrotik I linked to earlier, but that's not in my immediate plans. I should probably put Windows 7 on the old server before making it the backup server even though it's not on the internet which means I can just use a single port ConnectX-2 in the older system and a ConnectX-3 dual port in the new one. I'm favoring a 3 for the new one because it has PCIe 3.0 and only uses 4 lanes instead of 8 in a 3.0 motherboard.

 

[Handruin]

My servers are in the basement so getting a fiber or twinax to it would be a bit of work to route to my central network rack. I realized that the majority of network transfer occurs from my workstation on my second floor to the NAS in the basement so I'd benefit more from using 10GbE-BaseT than over fiber/twinax. I'm uncertain my CAT 6 lines can sustain 10Gb over the distance it was run (maybe 25 ft) but possibly.

I typically transfer large movies from an internal HDD which at most is capable of 180MB/sec with sustained reads. With 1Gb ethernet I'm consistently getting around 108MB/sec when transferring data so the jump to 180MB/sec isn't large enough to warrant a jump to a 10Gb network just yet (I know 10Gb is capable of much more). I realize I'd gain benefits from concurrent system access but it is so infrequent right now. I might be better off playing with bonding two CAT 6 cables since I have four in each room.

I've seen some 10Gb fiber switch show up on the used market and also found inexpensive SFP+ adapters from China that should work but the noise and power from that is a deterrent. At work we have a mixture of 10Gb fiber and 10Gb BaseT to play with. All the adapters I've encountered so far have been from Intel. Most of the work we use 10Gb for is iSCSI traffic since our product makes use of it heavily along with Fibrechannel.

 

[Stereodude]

My servers are in the basement so getting a fiber or twinax to it would be a bit of work to route to my central network rack. I realized that the majority of network transfer occurs from my workstation on my second floor to the NAS in the basement so I'd benefit more from using 10GbE-BaseT than over fiber/twinax. I'm uncertain my CAT 6 lines can sustain 10Gb over the distance it was run (maybe 25 ft) but possibly.

My servers and network rack are all in the basement, so it's less of a problem for me. I ran CAT6 all over my house. Monoprice sold it as 500MHz CAT6. 10GbE is rated for 50m on CAT6 and 100m on CAT6A. CAT6 spec only requires 250MHz. CAT6A requires 500MHz. I'm not sure if the Monoprice wire is effectively CAT6A despite not having been sold as CAT6.

For normal use I'm not moving big files from any of the computers in the house to the server in the basement. It's mostly just low data rates to support real time playback of media. The daily automated backup between the systems is where speed would help, but even then it's usually updating less than 200gB any given night. Some nights it doesn't copy a thing. The speed would really help during the initial backup or copying the data to the new server in the first place which for all intents and purposes is a one time event per machine.

And now I've pretty much talked myself out of this if I'm being logical... Unless I get an Intel card with XP x64 support this wouldn't help me get the data onto my new NAS/server since I need to have the new one up and running before I can put Windows 7 on the old one. At most it would help me with the big copy from the new one to the old one once it's got Windows 7 and is in its new role as the nightly backup server. My original plan was to plug the Perc 6i card with the 8 drive array connected into my new machine to copy the data from the old array to the new array. 10GbE doesn't help me tackle that task unless I get a card with XP x64 support.

 

[Stereodude]

Is it really as simple as buying two Mellanox Connectx-2 cards for under $20 a piece and a few meter Twinax SFP+ for under $20 and being in business?

Well, the answer is yes. I now have an Intel X520-DA1 in one Windows XP x64 system and a Mellanox ConnectX-3 in a Windows 10 system with 3m of Cisco Twinax / DAC connecting them. I'm not getting 10Gbs between them during file transfers, but I suspect that has more to due with Windows XP x64's TCP/IP stack and SMB implementation than the NiCs. It's still several times faster than anything I've ever seen from 1GbE and Windows XP x64.

 

[Chewy509]

I'm not getting 10Gbs between them during file transfers, but I suspect that has more to due with Windows XP x64's TCP/IP stack and SMB implementation than the NiCs. It's still several times faster than anything I've ever seen from 1GbE and Windows XP x64.

We noticed the same thing at work with Windows 7 to Windows 2012/Samba4/OpenIndiana, we would only get 150-200MB/s over SMB for a single transfer, despite all our tests showing 9.5Gbps+ available with iperf.
But what we noticed that for multiple transfers over SMB would could get near 8Gbps to/from the server, so it would seem that for single SMB connections it limits itself or has design limitations for a single transfer, can will scale if you initiate multiple connections/transfers.
Using NFS (Linux desktop to OpenIndiana server) demonstrated none of the performance issues as with SMB. (we would get near 900MB/s with NFS with TCP for a single file transfer).

 

[Stereodude]

We noticed the same thing at work with Windows 7 to Windows 2012/Samba4/OpenIndiana, we would only get 150-200MB/s over SMB for a single transfer, despite all our tests showing 9.5Gbps+ available with iperf.
But what we noticed that for multiple transfers over SMB would could get near 8Gbps to/from the server, so it would seem that for single SMB connections it limits itself or has design limitations for a single transfer, can will scale if you initiate multiple connections/transfers.
Using NFS (Linux desktop to OpenIndiana server) demonstrated none of the performance issues as with SMB. (we would get near 900MB/s with NFS with TCP for a single file transfer).

I've seen people post screenshots of >1GB/sec transfers so it's definitely possible. It seems to work better with two Windows boxes, and not if one of them is Linux.

I saw numbers as high as ~420MB/sec copying from my Windows XP x64 box today. Average was 300-400MB/sec for several hours until the box started recording some HDTV to another drive using a 1GbE interface. I'm not entirely sure why that slowed it down.

 

[Chewy509]

I've seen people post screenshots of >1GB/sec transfers so it's definitely possible. It seems to work better with two Windows boxes, and not if one of them is Linux.

I've seen a few of those screen shots, and some of them are very suspect. (Do the math: 10GbE, minus ethernet frame overhead, TCP/IP overheads, SMB/CIFS packet overhead...)

However, since neither Samba or the CIFS/SMB driver in illumos have SMB3 support, I would only expect you would only get maximal performance on Win8/2012 with SMB3 enabled.

 

[CougTek]

We noticed the same thing at work with Windows 7 to Windows 2012/Samba4/OpenIndiana, we would only get 150-200MB/s over SMB for a single transfer, despite all our tests showing 9.5Gbps+ available with iperf.
But what we noticed that for multiple transfers over SMB would could get near 8Gbps to/from the server, so it would seem that for single SMB connections it limits itself or has design limitations for a single transfer, can will scale if you initiate multiple connections/transfers.

So in essence, a single 10Gb connection behaves like a quad 2.5Gb etherchannel connection by default under Windows? Weird.

 

[Chewy509]

So in essence, a single 10Gb connection behaves like a quad 2.5Gb etherchannel connection by default under Windows? Weird.

For SMB2 connections, essentially yes. (SMB3 was added in Win8/Win2012, which added significant performance improvements).

 

[Handruin]

Not specifically targeting the SMB protocol but I'm actively seeing a sustained 8.2Gb/sec (1025MB/sec) during a DRBD block volume replication (sync) using a bonded pair of 10Gb Intel NICs (in round robin) via optical SFP+ direct connections under CentOS 6.7. The underlying drives are Intel P3600 NVMe SSDs (one per server).

 

[Stereodude]

So I ordered up two Mikrotik 24 port 1gig switches (CSS326-24G-2S+RM) yesterday for $105 each (shipped). That will give me 48 1GbE (Base-T) ports. I have more drops in my house than I have ports on my current 24 port 1GbE (Base-T) HP Procurve. That leaves me with some smaller gigabit switches chained off it. Each of the Mikrotik switch has two 10Gb SFP+ ports. I plan to connect the switches together using one SFP+ port on each with a short Twinax DAC cable. I will connect my main server to another of the SFP+ ports with another Twinax DAC cable. If I need more 10gig capability I can add their forthcoming CRS309-1G-8S+PC (8x SFP+ ports) or CRS312-4C+8XG (8x 10G/5G/2.5G/1G Ethernet ports & 4x 10G Ethernet / 10G SFP+ combo ports) depending on what I sort of 10Gb ports I need.

 

[Handruin]

Sounds like a nice setup. Are those switches noisy? Any word on the price point of the MikroTik CRS312-4C+8XG?

 

[Stereodude]

Sounds like a nice setup. Are those switches noisy? Any word on the price point of the MikroTik CRS312-4C+8XG?

The CSS326-24G-2S+RM are fanless, so they should be silent.

There aren't product pages yet for the two new Mikrotik's I mentioned. They were announced in their Newsletter #82, but no prices yet.

I'm looking to replace my HP Procurve PoE 10/100 switch also with something fanless. The Linksys LGS116P seems like a good candidate. I installed one at my last job to power the VOIP phones via PoE.

 

[Stereodude]

I got the two CSS326's today and installed them in my rack after work. My short Twinax DAC cable won't be here for a few more days so I used a 3m one I had from before to connect the two switches. My server is connected via 10g SFP+ only. Everything seems to be working and is happy.

 

[LunarMist]

Recently I've been looking at the QNAP switches. The QSW-1208-8C has 12 ports (8 combo SFP+/10GBASE-T) and the QSW-1204-4C has 8 (4 combo).
I can see that flexibility would come in handy with multiple devices.

They do have fans, but smart controlled. The worst case total power is 49W with all ports in use or 21W at idle.

 

[HubHacken]

No... it is my dream only! ))

 

[Stereodude]

Is there anyone who can school me on 10gig fiber?

I presume I just need a fiber based SFP+ module on each end. -SR ones will work for the lengths I'm looking at. Then, OM3 (or better) LC duplex fiber to connect the two right? I'd like to put a wall plate in the middle. It looks like they exist though.

 

[Handruin]

That should be all you need. The optical SFP+ transceiver can be picky with how they work in a given device. Are you buying SFP+ adapters that are from the manufacturer of the NIC and switch vendor? There are also some lower cost Chinese companies that make SFP+ for a lot of devices that claim support but I have no first-hand experience with them. I've seen incompatibilities in the 10Gb optical SFP+ among various vendors like Cisco and IBM. Often times you can find a compatible 3rd party transceiver but you'll have to do some searching or you can spend the top-dollar on the vendor-specific ones.

How far are you looking to run the connection? Can you not run a twinax DAC?

 

[Chewy509]

Mirroring what H mentioned, it's best to use the same vendor SFP+ transceivers as the card being used.

However we've had some good experiences with 3rd-party transcievers from FibreStore (https://www.fs.com/), as they make transceivers to suite vendor specific cards.

 

[Stereodude]

That should be all you need. The optical SFP+ transceiver can be picky with how they work in a given device. Are you buying SFP+ adapters that are from the manufacturer of the NIC and switch vendor? There are also some lower cost Chinese companies that make SFP+ for a lot of devices that claim support but I have no first-hand experience with them. I've seen incompatibilities in the 10Gb optical SFP+ among various vendors like Cisco and IBM. Often times you can find a compatible 3rd party transceiver but you'll have to do some searching or you can spend the top-dollar on the vendor-specific ones.

How far are you looking to run the connection? Can you not run a twinax DAC?

I'm thinking to connect my main desktop in my office/library to my switches in the basement via 10gig. AFAIK, it is too far for a twinax DAC. I'm guessing its around 50' total. I know 10GbE is potentially an option also (I ran 500MHz CAT6 when I wired everything up years ago), but I already have 10gig SFP+ NICs and switches with 10gig SFP+ ports. The 10GbE SFP+ modules are expensive. They also use more power and have worse latency.

Mirroring what H mentioned, it's best to use the same vendor SFP+ transceivers as the card being used.

However we've had some good experiences with 3rd-party transcievers from FibreStore (https://www.fs.com/), as they make transceivers to suite vendor specific cards.

From what I've read neither the Mikrotik switches or the Mellanox NICs I have care about the brand coding of the SFP+ modules. They sure didn't care about the brand coding of the DAC cables I've used.

 

[LunarMist]

I would see what speeds you get from the current cable unless you enjoy working on the fibers.

 

[Stereodude]

I would see what speeds you get from the current cable unless you enjoy working on the fibers.

Well, I don't have any hardware that supports copper 10gig, so that would be rather difficult. Fiber is pretty easy. You buy pre-terminated lengths and use them. One to the keystone in the wall plate from the switch and another much shorter one from the keystone to the PC. They fiber is surprisingly cheap. Like $8 for 15m for OM3. For the price I'd install OM4 to have future 100gig capable fiber installed since it's all of $12.

 

[Handruin]

Given the distance you need to run, OM3/OM4 makes sense. I wasn't familiar with your 10Gb switch so I was just describing the complications I've faced when dealing with some of the major switch vendors like Cisco, ibm, etc when using SFP+ optical. When using SFP+ DAC with the same hardware I do not have these issues. Most of my 10Gb runs at work are 5 meter or less so we use DAC. All the other optical is OM2/OM3 for fibre channel SAN connectivity.

Since the fiber cable is so cheap my only other recommendation would be to pull two of them. You should be good to go for many years and won't have to deal with EMI issues which is great.

 

[Stereodude]

It looks like a 7m OM4 cable will make the run I need in the basement from the switches into the wall. I think 6m would work, but they don't sell pre-made cables in 6m lengths, so 7m it is. 5m of OM4 should cover the in room portion. I reviewed my pre-drywall pictures of the wall and confirmed which stud cavities in the wall are unobstructed. I plan to run two additional RG6 cables between the two endpoints along with the fiber runs.

 

[Handruin]

Sounds like a good plan. What kind of traffic are you planning to manage that might use 10Gb?

 

[Stereodude]

What kind of traffic are you planning to manage that might use 10Gb?

I'm impatient when copying data to and from the server.

 

[Handruin]

That's as good a reason as anything else. Was just curious if you had a specific project that was limited by 1Gb.

 

[Stereodude]

That's as good a reason as anything else. Was just curious if you had a specific project that was limited by 1Gb.

FWIW, I did find that when doing mass FLAC to AAC (or MP3) conversion on my dual processor Xeon E5 v2 system that the 1gbE link was saturating and "starving" the processors, but I very rarely do that, and that's not the computer getting the 10gig NIC.

I ordered generic coded -SR transcievers and LC/LC OM4 fiber from Fiberstore.com yesterday. The 5m fiber cable I ordered is shipping from China despite the website saying it's in stock in the US. The rest is shipping from their US warehouse.

This morning I got an e-mail in Chinglish from someone at Fiberstore trying to be a little too friendly asking me about my company and what brand of networking equipment I was going to use it with so they could advise me of compatibility. I did a bit of research before I bought and based on everything I found the Mellanox cards don't care about coding, nor does the Mikrotik stuff so I just went generic.

 

[Handruin]

I'm definitely interested to know how things work out with Fiberstore SPF+ transceivers. I've read other people using them with success rather than pay the huge prices for the OEM ones.

At some point I will also want to dabble into some 10Gb SFP+ setups once I get things organized at my new place. I only have a few right now that are direct-connect between machines for replicating data.

 

[Stereodude]

I'm definitely interested to know how things work out with Fiberstore SPF+ transceivers. I've read other people using them with success rather than pay the huge prices for the OEM ones.

At some point I will also want to dabble into some 10Gb SFP+ setups once I get things organized at my new place. I only have a few right now that are direct-connect between machines for replicating data.

All the stuff arrived today. Apparently the shipping time from China and Delaware is the same. I don't know how they made any money shipping a $5.60 fiber optic cable from China using FedEx International Economy. I have the LC keystones already. Hopefully I can get it installed before the weekend is over.

 

[Stereodude]

I replaced a 3m DAC cable with a 2m OM4 cable + two of the -SR transceivers from Fiberstore. It all works. I have a 10g link over it. The Fiberstore module shows up in the Mikrotik like this:

(I edited out the Serial Numbers)