4w ago

FTTH upgrade - getting my LAN multi gig ready

It was recently announced that FTTH will soon (finally) be available in my market. The provider coming to town offers rates up to 8g.

I'm upgrading from DSL at <100mbps - really exciting! However I will then face a bit of an issue.

I self host many services over my DSL, and use custom firmware on my router. My DSL modem is in a transparent bridging mode. I like the flexibility and customizability this setup provides.

The new service includes a WiFi 7 router, but that means I'll also potentially be subject to all the weird things providers like to do, like adding backdoors, opening shared WiFi networks, force deploying different firmware, etc. Plus I won't be running any kind of service on the router itself, which I do have today (transparent proxy etc). The router I have today is not going to enable me to touch the peak bandwidth available.

What're the best options to upgrade LAN components so that I can support multi gig internal networking speeds, ensure my self hosted services all function normally, and I take advantage of the bandwidth the ISP upgrade offers? In your personal opinion, is it worth it to invest in upgraded lan components?

Anyone have experience converting from 1G LAN to 2.5 or even 10?

Do I really need 8G FTTH, of course not, but if I ever wanted to get the max out of it, what does that take?

27 comments

Going from 100 Mbps to even a gigabit, if you're self hosting, is going to be a huge difference. If you want my opinion, save yourself some money, go with the lowest speed over a gigabit and gradually buy equipment with the money you'd save compared to the 8 gigabit plan.
As for the router, can you either send a picture of it from the ISPs website or name the ISP? With 8 gig being the maximum, you're going to be on XGS PON and I have a hunch I know what equipment you're getting, but want to make sure I'm right.
- Agreed. Gigabit stuff is cheap, anything north of that is a bit... niche. And it's totally fine if parts of your network are still at gigabit if you ever switch to faster, you just won't get full speed on those parts.
  So yeah, start by making sure everything can do gigabit at least, and hold off until you need something faster. Chances are, prices and energy use for higher end equipment will come down over time, so by the time you need it, it'll be a lot cheaper.
  If you're going to run cable though, get the higher end stuff. You don't necessarily need fiber (that's a pain), but cables capable of 10gbit are relatively inexpensive and you should probably do that instead of sticking w/ 1gbit lines. However, if fiber isn't a giant pain, maybe that's worth it to you.
- Ezee fiber is the ISP. Eero max 7 WiFi. XGS PON agree, it is symmetrical
  
  Dang. Not the company I was hoping.
  If they're using an eero router, I'm going to assume you'll just have an ethernet cable from an ONT then into the router. Ask the installer if you need to use the eero or can you install your own router. That may alleviate some of your concerns.
  I work for an ISP and self host. I have more things in place to track my usage than any ISP would put just because I make myself the guinea pig for new equipment and want to know exactly what is happening. You will never use a full 8 gig (at least as of now, obviously in the future that will change). If the extra money isn't an issue do it, but if you can "girl math" the $30 price difference, stick with that for a year and spend the extra $360 you saved on multi-gig networking equipment, that's what I'd do.

Just a couple thoughts (I have a mix of 2.5Gb and 10Gb):
Mikrotik switches are a nice alternative to Unifi. Much less lipstick on the UI but reliable and fairly priced.
If possible, you'll probably want to use your own router rather than the all-in-one provided by the ISP. In my case, the router provided to me (Eero brand) did not even have a port fast enough for my service, and would have been an instant bottleneck.
Options for 10Gb-capable PCIe adapters (what you might put in your server or desktops) are more limited (at least they were when I transitioned a couple of years ago). Intel-based network adapters seem to require less effort to get working (driver-wise) vs. some of the other 10Gb / SFP+ capable adapters.
Finally, you are correct: nobody needs an 8Gb internet connection. Aside from well-seeded torrent file transfers, you will never reach that limit (and probably still never). And, you'll need an adequate storage backend to write that fast.
- nobody needs an 8Gb internet connection
  The fuck you say....LOL.
- Much less lipstick on the UI
  The new release added a bunch of lipstick. Still not as pretty as Unifi, but it's a pretty sizeable facelift.

I just got my 1 gbps fiber for my huge server. I found that with 125 MB/s network's capacity, you hit hardware limitations very quickly, HDD in particular. Before going beyond 1 gig, you should ask yourself if you really need more, as you will need to upgrade everything to high specs, from hard drives, cards, cables, router, vpn, to finally notice your network stays close to 5 % usage most of the time, and that the 8Gbps advertised are only reached using a speed test, only at 4 in the morning, and that your expensive upgrade is in reality barely used.
A good router is expensive, a good network card with a good transceiver is expensive, and you need to have many, 8Gbps is CPU hungry, SSDs are wearing fast, it's electric power hungry. This speed comes with a huge investment and heavy running costs.
I will stick with 1 gig for quite sometime I guess, it's far enough for my heavy usage, I can't imagine being too restricted to invest hundreds, if not a couple of thousands.

The new service includes a WiFi 7 router
I don't recommend it.
I would shoot for a 4 port 2.5Gbe unmanaged switch with 2 SFP+ ports (6 total ports) for 10G networking. 2.5Gbe is going to be more than enough for any WiFi solution you choose with room to upgrade 10G to WiFi if you wanted to spend a bit more on a higher tier WiFi router still leaving a single SFP+ port for 10G networking from your PC.
Biggest hit for your buck. Gonna set you back $40-50.
but if I ever wanted to get the max out of it, what does that take?
Kind of a lot. At least a top to bottom upgrade, from modem (PON), to 10G networking, to new Ethernet cables, to new 10G network drivers. Looking at a few hundred if you do it right. I also had Optimum's 8Gbps internet and was never able to even get anywhere near advertised speeds due to network saturation. IMO, the upgrade right now is too expensive to justify the expense for what you get. If you were confident you would be able to max out the connection, that would be a different story. But ultimately it's gonna be up to you. If you don't mind dropping a few hundred on upgrades, then go nuts.

Here's my opinion:
get 1G WAN - it's a huge upgrade, and you probably won't notice going much faster unless you're downloading/uploading a lot of stuff over the internet; it's probably substantially cheaper
consider 2.5G LAN, if it's not much more expensive than 1G - fast transfers over your LAN are much more likely to be noticed than transfers over the internet
put in CAT 6 cables at least, since that's capable of 10G in case you decide you want it; it's not much more expensive than 5E (1G capable), and then you won't need to redo it later; or better yet, run fiber everywhere, though that's more of a pain
Then upgrade anything that's <1G on your LAN, and leave the rest as-is until you actually need it. Chances are, you won't, and it's not worth spending the money. Prices for 2.5G and 10G (and higher) will eventually come down, so put it off until you actually need it and you'll probably save money in the long run.
In terms of what It takes, I think others gave good insight. Here's my basic summary:
expensive router and switch - copper can do 8G, but you'd probably want fiber if there's a chance your ISP will offer upgrades
start converting to SFP+, since that's likely what you'll want when upgrading things in the long run
some kind of mesh WiFi network - higher bitrates tend to be at higher frequencies, which have poor penetration; starting out w/ a mesh means it'll be easier to swap out APs as you increase bitrates/solve signal issues in various rooms
run lots of cable - the best mesh is one that's backed by cable
It's going to be expensive supporting anything over 2.5G in an entire network. Honestly, 1G is probably fine, and you can upgrade things more incrementally as you decide to improve speeds between endpoints (big ones are anything that handles high bitrate video).

Get a nice hardware capable of running opnSense and use that immediately after your new ISP device. Just ignore their WiFi router, it will be crap whatever it is, unless you cat reflash with OpenWRT.
Be prepared that the new ISP will .most probably have CG-NAT.
Note: opnSense is based on *BSD so make sure the hardware you buy has supported 10gb network cards, at least two.
- opnSense and use that immediately after your new ISP device
  This. It's how I set up my local network, tho I went with pFsense and then hung Pi-Hole+unbound off the managed switch. I did have to set up a vlan just for my lady friend because, apparently, screens full of ads is exactly what she likes, and I don't want to have to reinvent the wheel every time she comes over. So, I guess you could say it's pain avoidance. LOL
- Interesting point about cg nat. With my current ISP setup I get an actual (dynamic) ip4 address and ip6 thru 6rd. Can I still point my domain to the nat'd address?
  
  No, if you are batted, you will need a vps or some kind of real public address and tunnel to it for external to internal access. A VPN with port forwarding will also work.

Anyone have experience converting from 1G LAN to 2.5 or even 10?
Going from 1 G to 2.5 G is fairly cheap these days. You can almost certainly use the same cabling, even when you've got only Cat.5e cabling. While you can do 10 G over copper, I wouldn't suggest doing that, since it consumes quite a lot of power compared to both 1 G and 2.5 G. You'd need Cat.6E for reliable 10 G over copper.
- To be pedantic, there is no 6e. Just 6A. I am looking at a spool labelled 6e as I type this, but that’s just a manufacturer thing, not an actual spec.
  
  Oops, you are correct of course, 6A is what I meant, plain 6 should work fine also most of the time, but there is pretty much no point going for that, unless you have that deployed already.

Does the new ISP require use of their router or just offer it as an option?
AT&T used to require using their router, which was a pile of hot garbage. I have a Mikrotik Router and managed to mostly cut the AT&T router out but I had to configure my router to use the AT&T router for authentication, at which point the Mikrotik would take over. It was complicated to configure but it worked.
- At this stage I don't know - but they seem to describe a setup where they are essentially providing a WiFi 7 router as an access point, which connects to another piece of hardware acting as the gateway
  
  If they don't say it's required, assume it's not and ask them for details to run your own. IMO, you'll be happier if you can control exactly what you're running.

UniFi seem to have dabbled with 2.5 GBE briefly and then jumped to 10. I'm guessing that 10 will be the way to go.
You're looking at cat 6A patch leads rather than 7. 7 requires different but RJ45 compatible connectors, I believe. Yes, I'm still trying to understand what the difference is.
I have a 2.5G router, the CG Max. A 1 G switch (waiting for a reasonably priced 10 G) and a 10 G WAP. It's a bit of a mess!

Another point: 8gbps is mostly pointless. I would stay at 1gbps inside home and don't bother to rewire and replace all my home equipment. That's a long con game over the years slowly when each device has to be replaced anyway.
Maybe plan for 2.5gbps inside for the time being of you can do that a zero cost like reusing wiring.
I wouldn't count on WiFi in any case, at best it's a jimnick at that speed.

@cmeu What kind of custom firmware do you use? Is that #openwrt - if not consider it.
Check with your FTTN provider - Are they using CGNAT or any other IP4 sharing tech - if so you will need to see if they will do you a static IP or otherwise except you from IP sharing.
At least outside the US most ISPs will let you use your own router with their Fibre modem but if not you need to know how to get their device into bridge mode or at least forward some ports to your own devices.

27 comments