After struggling for over 20 hours, I wanted to share the results of my investigation regarding very poor Internet upload erformance.

Setup

  • Proxmox Server with a Single 10GbE NIC
  • OPNsense VM on Proxmox
  • OPNsense uses VirtIO NICs tied to the 10GbE Linux Bridge
  • upstream Gateway is a OpenWRT router with 1GbE uplink
  • Zyxel XS1930 Switch connecting Proxmox Host and Gateway

Problem

Internet download speeds are fine (900Mbit/s) but upload speeds are not (5-15MBit/s instead of 50MBit/s)

Solution

Various OPNsense tunables (configured for 8 CPU cores)

  • hw.ibrs_disable = 1
  • net.isr.maxthreads = -1
  • net.isr.bindthreads = 1
  • net.isr.dispatch = deferred
  • net.inet.rss.enabled = 1
  • net.inet.rss.bits = 6
  • kern.ipc.maxsockbuf = 16777216
  • net.inet.tcp.recvbuf_max = 4194304
  • net.inet.tcp.recvspace = 262144
  • net.inet.tcp.sendbuf_inc = 16384
  • net.inet.tcp.sendbuf_max = 4194304
  • net.inet.tcp.sendspace = 262144
  • net.inet.tcp.soreceive_stream = 1
  • net.pf.source_nodes_hashsize = 1048576
  • net.inet.tcp.mssdflt = 1240
  • net.inet.tcp.abc_l_var = 52
  • net.inet.tcp.minmss = 536
  • kern.random.fortuna.minpoolsize = 128
  • net.isr.defaultqlimit = 2048

Enabling Multiqueue in Proxmox for the VirtIO NICs

(binary stepping, 1 Queue for 2 cores, 2 Queues for 4 cores, 3 Queue for 8 cores ect, total amount of all Queues mustn’t be greater then the VMs CPU cores)

Enabling Flow Control on all involved Network devices

  • Proxmox hardware NIC: ethtool -K nic0 rx on tx on
  • OpenWRT lan interfnace:

uci set network.lan.txpause='1'

uci set network.lan.rxpause='1'

uci commit

reload_config

  • Zyxel Switch:

Port -> Port Setup - Checked all Ports

Enabling Port Buffering

Zyxel Switch:

Port -> Port Buffer - Checked the Port with the Gateway

Reason

The Main reason for this problem seems to be the down-stepping of 10Gbit traffic to 1Gbit devices. Without Flow control enabled on all involved devices, the sending rate can’t be adjusted. But without enabling Port Buffering, the Switch won’t allocate resources for adjusting the traffic flow rate for slower devices.

This Problem should only affect people who use devices with different link speeds on the same switch.

  • Yggstyle@lemmy.worldEnglish
    71·
    5 hours ago

    I find that outside of chasing diminishing returns - the biggest boost to upload speeds generally involves getting service that isn’t from spectrum or comcast.

    The speeds you are suggesting seem to imply you are on some cable carrier. Be consciously aware that there is frequently fine print stating up to on your meager upload allotment. You may be optimising against unseen forces.

    It looks like you put in some effort on tuning which is awesome - and hopefully is helpful to some as a starting point… but from experience tuning is often very unique to each individual setup. That said: kudos on drawing that much out of your setup.

    • frongt@lemmy.zipEnglish
      2·
      4 hours ago

      True, but if it’s only the proxmox host with the problem, it’s probably not the cable carrier. The traffic from your different devices looks mostly the same to them.

    • SmashOPEnglish
      2·
      4 hours ago

      Yep, still on cable… vodafone to be exact. I should get FTTH this year, but also then only 1000 down/500 up for almost double the price (and it’s Telekom, so cloudflare services are unusable).

      But the Problem I was facing was unique to the 10GbE Adapter. I used a 1GbE adapter before for years, without this issue.

      • Yggstyle@lemmy.worldEnglish
        3·
        4 hours ago

        But the Problem I was facing was unique to the 10GbE Adapter. I used a 1GbE adapter before for years, without this issue.

        Every home labber will eventually have that coming of age trial: ritual combat with the monster they have constructed. Glad to see you got the best of yours haha.

        • SmashOPEnglish
          1·
          3 hours ago

          Yep, I moved away from having multiple 1GbE NICs to a single 10GbE NIC and creating the network virtually using VLANs and SDNs. This freed up some PCI-E slots and cables, but also spawned a new boss