this post was submitted on 09 Jul 2024
84 points (92.0% liked)

Selfhosted

40006 readers
874 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
 

My old setup was:

VSDL modem -> pfsense on mini J1900 Celeron (2 GHz) -> CISCO SG300 10MPP switch -> Rukus R310 wifi -> Laptop

Currnet setup

Fiber model -> pfsense on mini J1900 Celeron (2 GHz) -> CISCO SG300 10MPP switch -> Rukus R310 wifi -> Laptop

Today i got my 1GBit fiber installed (big deal for those like me living in rural areas) only to discover that my current network setup is not allowing me to benefit from it.

I was on VSDL copper wire before and was probably in the region of 50-60 MBit/s with my above current setup. Even when removing the wifi bottle and linking with Cat5 UTP wire directly to switch, I'm not getting major improvements.

When I got the fiber installed this morning I was disappointed when I saw only marginal gain running at 80 MBit/s (c. +30 MBit). So I decided to connect the laptop via LAN cable directly to modem. I got a starkling 900MBit/s. So, along my network I have bottlenecks.

THe first one I tested was my little pfsense machine. I installed the speedtext-cli command and was surprised to find that it was giving my around 300 MBit/s. So a lot better than my laptop on its usual wifi connection but still only 33% of what I get directly off the modem.

So my first question is how can it be that my little mini J1900 Celeron (2 GHz) with 4 GB RAM cannot handle this bandwith? Do I need an upgrade for my pfsense machine? I noticed that the peak CPU demand as speedtest-cli was running was in the 60% region, far from a saturated CPU and RAM only occupied for about 30%. If it is my little pfsense machine, how far do I have to go with finding the right little machine that can handle 1 GBit/s.

The next question is if I'm getting 300 MBit/s on the WAN connection of the pfSense machine, how is it that I only see a small percentage of this on my laptop? i.e. a drop from 300 MBit/s to 80 MBit/s? I guess I would have to test the switch to start and then move to the wifi access points ...

top 44 comments
sorted by: hot top controversial new old
[–] [email protected] 69 points 4 months ago (2 children)

Have you checked all the ethernet links are actually connected at 1G and not 100M?

[–] [email protected] 22 points 4 months ago

FYI: Lots of the managed switches or the expensive wifi access points should be able to show the link status in their webinterfaces. It should be pretty easy to figure out if they're running at 100M. (Sometimes also some LEDs light up in a different color.)

[–] [email protected] 3 points 4 months ago

Yes, checked and are all on the 1000M (1G) link

[–] [email protected] 37 points 4 months ago (1 children)

how can it be that my little mini J1900 Celeron (2 GHz) with 4 GB RAM cannot handle this bandwith?

Because it's ancient, and when it was new it was bottom-of-the-barrel.

[–] [email protected] 2 points 4 months ago

I probably didn't realise how CPU intensive the work of 1Gbit connection must be ...

[–] [email protected] 30 points 4 months ago (1 children)

The question is what you do with your pfsense. IDS/IPS are quite CPU hungry and Celerons are not really fast CPU’s.

[–] [email protected] 1 points 4 months ago (1 children)

peak CPU demand as speedtest-cli was running was in the 60% region, far from a saturated CPU and RAM only occupied for about 30%

It doesn't look like he's bound by CPU.

[–] [email protected] 6 points 4 months ago (1 children)

And he is currently at 1/3 of the potential speed and 3*60% = 180% CPU load for 1Gbits. So I wouldn’t even bother troubleshooting further when you already know the hardware will be an issue sooner or later.

[–] [email protected] 5 points 4 months ago (1 children)

That assumes that all of the 60% is for pushing packets, which is almost certainly not the case.

[–] [email protected] 1 points 4 months ago

True. But since OP is using a benchmark anyways, I don‘t know how close to real world that is. If they are doing lots of filesharing, let‘s say with P2P networks, it could be way worse because of the number of connections. So I agree with you - I was just working with the info I had :)

[–] [email protected] 22 points 4 months ago (1 children)

So my first question is how can it be that my little mini J1900 Celeron (2 GHz) with 4 GB RAM cannot handle this bandwith?

  • check ethtool for link speed: sudo ethtool enp2s0 | egrep 'Speed|Duplex' Your device name may be different from enp2s0. use ip link to see all devices. if it's not
Speed: 1000Mb/s
Duplex: Full

then that's probably a bad sign.

  • that is a 10 year old celeron processor. celeron were the budget (a.k.a. cheapest, slowest) class processor at the time. it's quite likely that it cannot keep up.
  • If you still think it's not CPU directly, use iotop to see if you have I/O bottleneck.
[–] [email protected] 2 points 4 months ago (1 children)

sudo ethtool enp2s0 | egrep 'Speed|Duplex'

Assuming you mean running these in the command prompt of pfSense? Tried but says "not found". Same for "ip link"

[–] [email protected] 3 points 4 months ago (1 children)

pfSense is UNIX-based and those commands are generally included with Linux and probably Linux-specific.

[–] [email protected] 3 points 4 months ago

Ah, didn't realize pfSense is the OS, not something that runs on linux. My command examples won't work for you.

[–] [email protected] 15 points 4 months ago

Something to look for besides bandwidth is actual packet routing throughput. It's possible you enabled a feature (ex. Deep packet inspection) that is limiting how many packets can be routed per second given the speed of your hardware.

[–] [email protected] 14 points 4 months ago (1 children)

What can the network cards support?

[–] [email protected] 11 points 4 months ago (1 children)

Yes, e.g. rpi3b+ has gigabit ethernet, but it's only 300Mbit, because it's connected via usb2 internally. Something similar can be the culprit here as well.

[–] [email protected] 1 points 4 months ago

Check what drop your get connecting the wifi modem directly to the router. There is usually a massive drop from wired to wifi.

[–] [email protected] 11 points 4 months ago (2 children)

That Pentum is a budget CPU from just over 10 years ago. It has PCIe 2.0. Maybe the "gigabit" ethernet is connected to the CPU by a single 500Mbit PCIe lane.

[–] [email protected] 8 points 4 months ago

PCIe 2.0 is 500 MB/s per lane, it's not going to limit the speed. That CPU certainly doesn't have enough power to run something heavy like IDS at 1gbps though.

[–] [email protected] 7 points 4 months ago

500MB, not Mb. Order of magnitude difference there.

[–] [email protected] 9 points 4 months ago

Is the laptop connected to 2.4 or 5 GHz Wirth WiFi? In my setup I never get more than 80 Mbit with 2.4 GHz.

[–] [email protected] 8 points 4 months ago* (last edited 4 months ago) (1 children)

Do yuo have IDP/IPS turned on on pfSense? My OPNsense on my 1Gbps fibre will easily drop from an average of 900Mbps down to around 300Mbps-500Mbps, if I turn on IDS.

[–] [email protected] 1 points 4 months ago

I dont' have IDS/IPS installed on my pfsense box.

[–] [email protected] 6 points 4 months ago (1 children)

By the way OP, similar but worse is the ability to handle 25Gbits. But someone made a working router for that as well and CPU was also a factor: https://michael.stapelberg.ch/posts/2022-04-23-fiber7-25gbit-upgrade/

[–] [email protected] 0 points 4 months ago (1 children)

I don't understand why someone would need 25GbE

[–] [email protected] 1 points 4 months ago

Because it's dope.

Also, according to their website the 10 and 25 Gbit/s packages cost the same per month.

Also, still cheaper than my 1 Gbit/s connection.

[–] [email protected] 6 points 4 months ago

Another piece of the puzzle is probably your WiFi router, as you normally won't get speeds near 1Gbps over WiFi. In order to benefit maximally from it, you need to connect your devices (laptops, stationary PC, TV, etc.) with a cable to get the most of it.

You should also try to disable some pfSense plugins, like OpenVPN, zenArmor, etc. as they will severely limit your bandwidth throughput. But as others said, most likely you will also need to upgrade your hardware box, and you can migrate to OPNsense while at it.

[–] [email protected] 6 points 4 months ago

Might be pfsense? My little J1900 box running Ubuntu Server gets 920Mbps.

[–] [email protected] 5 points 4 months ago* (last edited 4 months ago)

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters More Letters
AP WiFi Access Point
NAT Network Address Translation
PCIe Peripheral Component Interconnect Express
PoE Power over Ethernet
VPN Virtual Private Network

5 acronyms in this thread; the most compressed thread commented on today has 11 acronyms.

[Thread #861 for this sub, first seen 9th Jul 2024, 21:35] [FAQ] [Full list] [Contact] [Source code]

[–] [email protected] 4 points 4 months ago (1 children)

I have been using similar hardware setups in A LOT of installations. This is mostly an issue with the pfSense hardware. There's a lot of decent options around $200, mostly focus on getting a modern CPU (if it supports AES-NI there's a good chance it will be fine). A lot of them have 2.5g nics these days too.

Sadly, sg300 line is also getting pretty old. In recent years I'm seeing more and more issues with them, especially in the models with poe. Sg350 is even eos now, with cbs350 being the current.

I'm also now using r610 as the absolute minimum ruckus ap. I was using r510 for a long time, but the r610 has noticable improvements.

[–] [email protected] 1 points 4 months ago

Any thoughts on a good little fanless device that I can use as a pfSense machine that has a resonable CPU. I would just swap the SSD from my curent device to the new one and it should all work nicely.

[–] [email protected] 4 points 4 months ago (1 children)

For what it's worth, since it sounds like you will be hardware shopping soon: I am using a 2.4GHz Intel Atom C2758 running pfSense and get 2Gb/s down and around 1.5Gb/s up through it. I am using an add-on Intel-based PCIe network adapter, so I'm not sure if that is helping with the CPU load. But it works well.

[–] [email protected] 2 points 4 months ago

Any more specific recomendation of the machine you have running?

[–] [email protected] 3 points 4 months ago (1 children)

I had to upgrade my pfsense hardware when I got fiber several years ago, which was in a similar situation as yours. The CPU just couldn't handle the connection table.

[–] [email protected] 2 points 4 months ago

J1900 has no hardware switch. Every packet goes through CPU, so even LAN to LAN uses processing power. Add pfsense to the mix and it's probably choking.

[–] [email protected] 2 points 4 months ago* (last edited 4 months ago)

Maybe look into hardware offloading

What's your ram speed?

[–] [email protected] 2 points 4 months ago (1 children)

The process is to go step-by-step. First direct connect to modem you have, bridged connection if possible, and test with multiple bandwidth measurements (speedtest, fast.com, downloading a big file from some university ftp...) and work your way downstream of the network. And on every step test multiple scenarios where it's possible, preferably with multiple devices.

When I got a 1Gbit fiber connection few years back I got an Ubiquiti Edgerouter-X with PoE-options. On paper that should've been plenty for my network, but in theory with NAT, DNAT, firewall rules and things like that it capped on 6-700Mbps depending on what I used it for. With small packets and VPN it dropped even more. So now that thing acts as an glorified PoE switch and the main routing is handled with Mikrotik device, which on manufacturers tests should be able to push 7Gbps on optimal conditions. I only have 1/1Gbps, so there's plenty of room, but with very specific loads that thing still is still pushed to the limit (mostly small packet size with other stuff on top of it) but it can manage the full duplex 1000Base-T. And on normal everyday use it's running at 20% (or so) load, but I like the fact that it can manage even the more challenging scenarios.

[–] [email protected] 1 points 4 months ago (2 children)

Ok, starting to think I need a new little device for my pfSense. I was thinking of going OpenSense and buying one of their devices to support the project.

Regarding my switch, the ports where my Rukus APs are connected are showing 1000M on the interface. But I think a step by step testing is what is needed as suggested above.

[–] [email protected] 3 points 4 months ago

Run iperf internally to see if your bottleneck is switch/ap or fw. I set up a j1900 pfsense for my sisters family a while back to do qos (gamer bois in the house) amd it had no problem staying at 500mbps. No ids or other stuff.

Not built any opn/pf-sense in a while, but i always use intel server-nic’s. Used to have way better support than other stuff on bsd

[–] [email protected] 1 points 4 months ago (1 children)

They are expensive but I run a OPNsense DEC740 and have no issues with my Gigabit fiber, even without modem and the PPPoE overhead.

You can still try playing with hardware offload on/off and if you use PPPoE, it runs on a single core by default.

[–] [email protected] 2 points 4 months ago (1 children)

DEC740

Very nice but looks expensive. Do you think I could upload the pfSense configuration to it? I dread the pain of having to configure the whole thing from scratch.

[–] [email protected] 1 points 4 months ago

I don't think you can import pfSense configurations into OPNsense. I switched from a DIY pfSense box as well and redid the config.

You can look for a converter or install pfSense onto it though.

[–] [email protected] 2 points 4 months ago

CPU and RAM are not the only limiting factors. Not only that but not everything runs multithreaded. Maybe some piece of the puzzle is not multithreaded and is using all it can from a single core (assuming that cpu is multi- core)

Depending on how much you value your time, you're almost certainly better off getting a new machine to run pfsense.