After struggling for over 20 hours, I wanted to share the results of my investigation regarding very poor Internet upload erformance.

Setup

  • Proxmox Server with a Single 10GbE NIC
  • OPNsense VM on Proxmox
  • OPNsense uses VirtIO NICs tied to the 10GbE Linux Bridge
  • upstream Gateway is a OpenWRT router with 1GbE uplink
  • Zyxel XS1930 Switch connecting Proxmox Host and Gateway

Problem

Internet download speeds are fine (900Mbit/s) but upload speeds are not (5-15MBit/s instead of 50MBit/s)

Solution

Various OPNsense tunables (configured for 8 CPU cores)

  • hw.ibrs_disable = 1
  • net.isr.maxthreads = -1
  • net.isr.bindthreads = 1
  • net.isr.dispatch = deferred
  • net.inet.rss.enabled = 1
  • net.inet.rss.bits = 6
  • kern.ipc.maxsockbuf = 16777216
  • net.inet.tcp.recvbuf_max = 4194304
  • net.inet.tcp.recvspace = 262144
  • net.inet.tcp.sendbuf_inc = 16384
  • net.inet.tcp.sendbuf_max = 4194304
  • net.inet.tcp.sendspace = 262144
  • net.inet.tcp.soreceive_stream = 1
  • net.pf.source_nodes_hashsize = 1048576
  • net.inet.tcp.mssdflt = 1240
  • net.inet.tcp.abc_l_var = 52
  • net.inet.tcp.minmss = 536
  • kern.random.fortuna.minpoolsize = 128
  • net.isr.defaultqlimit = 2048

Enabling Multiqueue in Proxmox for the VirtIO NICs

(binary stepping, 1 Queue for 2 cores, 2 Queues for 4 cores, 3 Queue for 8 cores ect, total amount of all Queues mustn’t be greater then the VMs CPU cores)

Enabling Flow Control on all involved Network devices

  • Proxmox hardware NIC: ethtool -K nic0 rx on tx on
  • OpenWRT lan interfnace:

uci set network.lan.txpause='1'

uci set network.lan.rxpause='1'

uci commit

reload_config

  • Zyxel Switch:

Port -> Port Setup - Checked all Ports

Enabling Port Buffering

Zyxel Switch:

Port -> Port Buffer - Checked the Port with the Gateway

Reason

The Main reason for this problem seems to be the down-stepping of 10Gbit traffic to 1Gbit devices. Without Flow control enabled on all involved devices, the sending rate can’t be adjusted. But without enabling Port Buffering, the Switch won’t allocate resources for adjusting the traffic flow rate for slower devices.

This Problem should only affect people who use devices with different link speeds on the same switch.

  • non_burglar@lemmy.worldEnglish
    5·
    6 hours ago

    Wow, you diagnosed buffer bloat and applied the fix to your LAN side? Sooo much work…

    The problem is unlikely to have been on the proxmox side. Multiqueue only allows virtio to multithread TCP connections via the host CPU using more than one virtual cpu, but this is essentially like aggregating a network link; it will increase bandwidth, but not throughput. Besides, the actual limit for the proxmox internal bridge and virtio NICs is “whatever the cpu can manage”, which is sometimes over 10Gb. It’s unlikely to be slowing down traffic coming from your vms.

    • SmashOPEnglish
      2·
      6 hours ago

      Yeah, two whole days of work 😅 The alternative would have been to install a dedicated 1GbE NIC in the servers again. The tunables and proxmox settings probably don’t do anything now but maybe in the future when I finally get FTTH. I read a lot about OPNsense performance optimization and these tweaks shouldn’t hurt anything, so might as well apply them for good measure.

      • non_burglar@lemmy.worldEnglish
        1·
        2 hours ago

        Well, seems like it was time well spent in any case.

        If you have classic upstream buffer bloat, there are a couple of traffic shaping algorithms (cake and fq_codel) that work really well with the majority of competent routers, including opnsense/pfsense.

        Traffic shaping is definitely a can of worms, but fun to learn.