5GbE

[LunarMist]

I'm seeing as bit more of the 5Gb lately. Is it becoming the new standard?

 

[Mercutio]

It exists and it works over vanilla twisted Cat 6 pair over modest distances and adapters work on x1 PCIe or vanilla USB 3, but you'd need to use it with what is probably a more expensive switch. I'm not even fully convinced 2.5GbE is a "standard" because it's just this half-assed option for consumer hardware, but it does work and it is faster than 1GbE, so who am I to complain?

 

[Handruin]

I've been aware of 5Gb but haven't encountered it yet in any hardware I've worked on. To Merc's point, finding a compatible and affordable switch is going to be the challenge. Probably easier to just jump to 10Gb over cat 6A.

 

[LunarMist]

Thanks. I'll just buy 2.5GbE gear if replacement with 5GbE is a ways off yet.

 

[ddrueding]

I also think that 2.5 is a bit of a hack, with 5 even more so. I suspect most installs will go from 1 to 10 directly. Considering how much 10Gb gear is already out there, and how easy it is to work with, I'm surprised there is this much effort of half-steps.

 

[LunarMist]

I'm not seeing 10GbE in any of the common chipsets. It seems to be an add-on to the motherboards through PCIe. Where else are they?

 

[ddrueding]

The HEDT type boards, and the blingy ones can have them, here is one I was considering for my current build.

ASRock Z690 AQUA OC

Supports 14th, 13th & 12th Gen Intel Core™ Processors (LGA1700); 20 Phase SPS Dr.MOS Power Design; 2 x DDR5 DIMMs, Supports Dual Channel, up to 7000+ (OC)<span style=color:red;>*</span>; 2 PCIe 5.0 x16, 1 PCIe 4.0 x16, 1 PCIe 3.0 x1; Graphics Output Options: HDMI, 2 Thunderbolt™ Type-C<span...

www.asrock.com

 

[LunarMist]

That's what prompted my considerations, but 5GbE could be a while for the mainstreamsb if InTel and AMD do not support it.

 

[Mercutio]

I think a big reason why 10GbE hasn't entered consumer mainstream is that single ports of 10GbBaseT use crazy amounts of power, like 6 - 7W/port. I don't know if 5GbE has the same problem to the same degree or not. I am assuming that it does. 10Gb twisted pair also suffers terribly from attenuation. Cat 6 runs also have to be fairly short (~50 feet) to maintain 10Gb speeds.

 

[Mercutio]

Neither of the server-class 10Gb NICs I have here will negotiate a 2.5GbE connection on Cat6a with this cheapo 2.5GbE switch I just got.
But it did work with a $10 PCIe NIC with a Realtek chip on it and my Asus 2.5Gb USB dongle. I'm not surprised that Intel and Broadcom didn't bother to implement in-between non-standard Ethernet revisions, but it would've been cool if they did.

 

[LunarMist]

From what I read, 2.5 and 5GbE are much newer standards than 10GbE so are not supported by many products that came out before then. When 10GbE came out it was mostly for datacenter and backroom stuff at businesses, not for individual machines, right?

 

[Mercutio]

I don't think 2.5 or 5GbE are formal standards at all, but in theory I think they're just derivatives of 10Gb anyway. I'm pretty sure the basic functionality could be added in drivers if there was real demand.

Something weird I've observed with the Realtek 2.5Gb NIC is that SMB connections are fast but have weirdly high latency. I'm not terribly motivated to fix it since it's on a gaming computer, but I can't do smooth playback of a 4k gameplay recording. It was a $9 card though. I'm assuming it's offloading everything to the CPU and using interrupts instead of direct memory access or something equally boneheaded.

 

[LunarMist]

I would not be surprised. Have you tried the Intel 226V?
It looks like there are standards since 2016. https://en.wikipedia.org/wiki/2.5GBASE-T_and_5GBASE-T
Who knows what devices truly comply.

 

[Mercutio]

I would not be surprised. Have you tried the Intel 226V?

Not yet. I actually got the NICs I have last time I was at Microcenter. The original idea was just to be able to direct connect the USB and the PCIe guy to speed up new setups on laptops.

 

[Mercutio]

Here's a 10 port Mikrotik SFP/SFP+ L3 switch that can handle 10GbE on 9 ports that consumes a very reasonable 20W under full load. Most rack mountable 10Gb switches are more like 200W and sound like dying wildebeests. It's $180 on Amazon.

 

[Handruin]

The way I read that switch spec, it only supports 4 x 10Gb ports since only 4 of them are designated as SFP+ and the others are limited to SFP.

It still looks like a decent switch.

 

[Handruin]

The path I took for 8 x 10Gb was to go through this thread regarding the Brocade ICX7250 and bought one on eBay for around $300.

I ended up getting the 48 port version (with 8 x 10Gb SFP+) and upgraded the firmware to the latest Ruckus version outlined in their docs and unlocked the licensed ports.

The 48 port version is a 65W switch (24 port is 50W) and can confirm the fan is near silent after the switch boots and loads the firmware for temperature management.

 

[Mercutio]

That's an interesting device. I wonder if that thing will talk to ConnectX cards in ethernet mode over the QSFP+ ports.

Edit: Dammit I was looking at the specs on the next model up.

 

[Mercutio]

I found a working 24 port 7250 with the 10Gb ports on Ebay for under $80 shipped. I'll have to pick up some AOC cables (it's hard to find twinax longer than 7m) to really mess with it but I definitely have a few 10GbE cards already. I wouldn't mind being able to attach a few more computers to big-boy bandwidth.

 

[Handruin]

Nice, that's an awesome price! There weren't a ton on the market when I got mine in 2022 so I paid a bit over $300 for it. Hope it works well for you.

One small annoyance with this 7250 is the serial console cable connection on the front is a ridiculous use of USB mini in a completely non-standard way. You can hack up an old cable or search for something like a "Brocade ICX7750, ICX7450 ICX7250 DB9 Female to USB Mini Serial Console Cable 6FT". I bought one just to make life easier when debugging and setting things up via my laptop. Once the mgmt port is setup you can just SSH or even use the basic web interface to configure it.

That forum thread I linked to earlier has instructions on how to setup a local TFTP server to host the firmware so that you can flash the firmware to the latest and also how to license all the ports if yours doesn't come enabled.

Note that it may sound super loud on first boot until the firmware loads which has the temperature management. Might take 60-90 seconds until the fans ramp down.

 

[Mercutio]

I'll just add that to the other $140 worth of cables I need to buy.
I'm actually surprised at how reasonable fiber is now. I remember paying $75/m for ST-ST patch cables back in the day and here I am getting just over 60m of cables for not quite double that.

 

[Mercutio]

I got my 7250. It only has four SFP+ ports on it, but that's more 10Gb ports than I have 10Gb NICs. My AOC cables are on a literal slow boat from China. "Arrival expected before December 6th." I ordered them two weeks ago. This is the price I pay to save $50 on cables.
It does power on and seems to be just fine for 1Gb switching, so at the very least I have a moderately priced switch I could stick in my colo if I find nothing else to do with it.

 

[Handruin]

Interesting I didn't notice they made a 4 port SFP+ version of the 7250. That may explain the much lower price compared to my 48 port + 8 SFP+

 

[Mercutio]

It looks like the ports 5 - 8 are blocked off with some kind of plastic cover. It didn't just pop off, so I'm assuming it's there for a reason.

 

[Mercutio]

I had to look at a picture of another Brocade to see. This thing had a (custom?) 3D printed cover fitted over the unoccupied ports. The occupied ports appear to have MTRJ transceivers in them, a fiber connector I've never actually seen in use before.

 

[Mercutio]

I finally got my new cables and fired up the 7250. The damned thing is full of black particulates all over the inside. I'm going to have to replace the fans. At least that explains the cover over the SFP ports and probably the $75 price tag.