If you self-host all the same services you have the same exposure level if root on your hosting machine is compromised. I suppose it depends on how confident you feel in how agile you can patch if a vulnerability becomes known in postfix for example. I wouldn’t consider self hosting something that reduces your cybersecurity risk typically
The Pi4 USB controller and network adapter share bandwidth. Do you have any devices on the USB port that could be causing collisions? I really can’t think of anything in that kind of scenario that would cause that sort of issue unless somehow you were using USB for video out…