5GbE

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
17,489
Location
USA
I'm seeing as bit more of the 5Gb lately. Is it becoming the new standard?
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,250
Location
I am omnipresent
It exists and it works over vanilla twisted Cat 6 pair over modest distances and adapters work on x1 PCIe or vanilla USB 3, but you'd need to use it with what is probably a more expensive switch. I'm not even fully convinced 2.5GbE is a "standard" because it's just this half-assed option for consumer hardware, but it does work and it is faster than 1GbE, so who am I to complain?
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,918
Location
USA
I've been aware of 5Gb but haven't encountered it yet in any hardware I've worked on. To Merc's point, finding a compatible and affordable switch is going to be the challenge. Probably easier to just jump to 10Gb over cat 6A.
 

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
17,489
Location
USA
Thanks. I'll just buy 2.5GbE gear if replacement with 5GbE is a ways off yet.
 

ddrueding

Fixture
Joined
Feb 4, 2002
Messages
19,723
Location
Horsens, Denmark
I also think that 2.5 is a bit of a hack, with 5 even more so. I suspect most installs will go from 1 to 10 directly. Considering how much 10Gb gear is already out there, and how easy it is to work with, I'm surprised there is this much effort of half-steps.
 

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
17,489
Location
USA
I'm not seeing 10GbE in any of the common chipsets. It seems to be an add-on to the motherboards through PCIe. Where else are they?
 

ddrueding

Fixture
Joined
Feb 4, 2002
Messages
19,723
Location
Horsens, Denmark
The HEDT type boards, and the blingy ones can have them, here is one I was considering for my current build.

 

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
17,489
Location
USA
That's what prompted my considerations, but 5GbE could be a while for the mainstreamsb if InTel and AMD do not support it.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,250
Location
I am omnipresent
I think a big reason why 10GbE hasn't entered consumer mainstream is that single ports of 10GbBaseT use crazy amounts of power, like 6 - 7W/port. I don't know if 5GbE has the same problem to the same degree or not. I am assuming that it does. 10Gb twisted pair also suffers terribly from attenuation. Cat 6 runs also have to be fairly short (~50 feet) to maintain 10Gb speeds.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,250
Location
I am omnipresent
Neither of the server-class 10Gb NICs I have here will negotiate a 2.5GbE connection on Cat6a with this cheapo 2.5GbE switch I just got.
But it did work with a $10 PCIe NIC with a Realtek chip on it and my Asus 2.5Gb USB dongle. I'm not surprised that Intel and Broadcom didn't bother to implement in-between non-standard Ethernet revisions, but it would've been cool if they did.
 

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
17,489
Location
USA
From what I read, 2.5 and 5GbE are much newer standards than 10GbE so are not supported by many products that came out before then. When 10GbE came out it was mostly for datacenter and backroom stuff at businesses, not for individual machines, right?
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,250
Location
I am omnipresent
I don't think 2.5 or 5GbE are formal standards at all, but in theory I think they're just derivatives of 10Gb anyway. I'm pretty sure the basic functionality could be added in drivers if there was real demand.

Something weird I've observed with the Realtek 2.5Gb NIC is that SMB connections are fast but have weirdly high latency. I'm not terribly motivated to fix it since it's on a gaming computer, but I can't do smooth playback of a 4k gameplay recording. It was a $9 card though. I'm assuming it's offloading everything to the CPU and using interrupts instead of direct memory access or something equally boneheaded.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,918
Location
USA
The way I read that switch spec, it only supports 4 x 10Gb ports since only 4 of them are designated as SFP+ and the others are limited to SFP.

It still looks like a decent switch.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,918
Location
USA
The path I took for 8 x 10Gb was to go through this thread regarding the Brocade ICX7250 and bought one on eBay for around $300.

I ended up getting the 48 port version (with 8 x 10Gb SFP+) and upgraded the firmware to the latest Ruckus version outlined in their docs and unlocked the licensed ports.

The 48 port version is a 65W switch (24 port is 50W) and can confirm the fan is near silent after the switch boots and loads the firmware for temperature management.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,250
Location
I am omnipresent
That's an interesting device. I wonder if that thing will talk to ConnectX cards in ethernet mode over the QSFP+ ports.

Edit: Dammit I was looking at the specs on the next model up.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,250
Location
I am omnipresent
I found a working 24 port 7250 with the 10Gb ports on Ebay for under $80 shipped. I'll have to pick up some AOC cables (it's hard to find twinax longer than 7m) to really mess with it but I definitely have a few 10GbE cards already. I wouldn't mind being able to attach a few more computers to big-boy bandwidth.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,918
Location
USA
Nice, that's an awesome price! There weren't a ton on the market when I got mine in 2022 so I paid a bit over $300 for it. Hope it works well for you.

One small annoyance with this 7250 is the serial console cable connection on the front is a ridiculous use of USB mini in a completely non-standard way. You can hack up an old cable or search for something like a "Brocade ICX7750, ICX7450 ICX7250 DB9 Female to USB Mini Serial Console Cable 6FT". I bought one just to make life easier when debugging and setting things up via my laptop. Once the mgmt port is setup you can just SSH or even use the basic web interface to configure it.

That forum thread I linked to earlier has instructions on how to setup a local TFTP server to host the firmware so that you can flash the firmware to the latest and also how to license all the ports if yours doesn't come enabled.

Note that it may sound super loud on first boot until the firmware loads which has the temperature management. Might take 60-90 seconds until the fans ramp down.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,250
Location
I am omnipresent
I'll just add that to the other $140 worth of cables I need to buy.
I'm actually surprised at how reasonable fiber is now. I remember paying $75/m for ST-ST patch cables back in the day and here I am getting just over 60m of cables for not quite double that.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,250
Location
I am omnipresent
I got my 7250. It only has four SFP+ ports on it, but that's more 10Gb ports than I have 10Gb NICs. My AOC cables are on a literal slow boat from China. "Arrival expected before December 6th." I ordered them two weeks ago. This is the price I pay to save $50 on cables.
It does power on and seems to be just fine for 1Gb switching, so at the very least I have a moderately priced switch I could stick in my colo if I find nothing else to do with it.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,918
Location
USA
Interesting I didn't notice they made a 4 port SFP+ version of the 7250. That may explain the much lower price compared to my 48 port + 8 SFP+
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,250
Location
I am omnipresent
It looks like the ports 5 - 8 are blocked off with some kind of plastic cover. It didn't just pop off, so I'm assuming it's there for a reason.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,250
Location
I am omnipresent
I had to look at a picture of another Brocade to see. This thing had a (custom?) 3D printed cover fitted over the unoccupied ports. The occupied ports appear to have MTRJ transceivers in them, a fiber connector I've never actually seen in use before.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,250
Location
I am omnipresent
I finally got my new cables and fired up the 7250. The damned thing is full of black particulates all over the inside. I'm going to have to replace the fans. At least that explains the cover over the SFP ports and probably the $75 price tag.
 
Top