For public facing only use key based authentication. Passwords have too much risk associated for public facing ssh

so all rules would have to be applied on the Pi itself

Sounds like you’ll want to setup IPTables

Huh! Thank you very much for the detailed answer that’s extremely interesting!

Checking nmap’s documentation it looks like it’s perfectly possible to detect open udp ports

You should NOT have a WG tunnel from the home network to the VPS with fully unrestricted access to everything.

This is what I came here to make sure was said. Use your firewall to severely restrict access from your public endpoint. Your wiregaurd tunnel is effectively a DMZ so firewall it off accordingly

The really nice thing about tailscale for accessing your hosted services is absolutely nothing can connect without authentication via a professionally hosted standard authentication, and there’s no public ports for script kiddies to scan for, spot and start hammering on. There’s thousands of bots that do nothing but scan the internet for hosted services and then try to compromise them, so not even showing up on those scans is a good thing.

For example, I have tailscale on my Minecraft server and connect to it via tailscale when away from home. If a buddy wants to join I just send a link sharing the machine to them and they can install tailscale and connect to it normally. If for some reason buddy needs to be cut off, I can just stop sharing to that account on Tailscale and they can no longer access the machine.

The biggest challenge of tailscale is also it’s biggest benefit. Nothing can connect without connecting through the tailscale client, so if my buddy can’t/won’t install tailscale they can’t join my Minecraft server

No problem! I’m just an information sponge and I’ve lucked out with really good mentors so far in my career to learn from

So from my experience you generally will have different zomes of security. Outside Internet is obviously entirely untrusted so block every incoming connection except those you really need, and even then ideally all remain blocked (especially for a home network). Then you generally have your guest network which might need access to some hosted resources but is largely just used by guests to connect to the internet, next is your client network where your computer likely lives which probably gets access to all hosted resources but no management access (or depending on how much you want to trust your primary PC, limit that to just your main PC) and finally your datacenter network where you hopefully trust everything running in there.

You generally work with these zones and write rules based on the zone the traffic is coming from, with some exceptions, such as I might not want to give the guest network any access to my data center network, except for access to my jellyfin so I’ll create a rule allowing only tcp web traffic from that network to a specific port on a specific IP/hostname.

A common way to achieve this is with a DMZ network, a network that sits between all of your networks and relies heavily on routing and firewalls. Public services and routers get IP addresses on the DMZ, and your firewall only allows specific paths. The outside Internet can open connections to the web ports of the web server and nothing else, the web server can’t open connections to your other networks, only specific machines/networks are allowed to access the SSH port of the web server, etc. the DMZ is where trusted and untrusted connections mix, hence why its named after the zone that belongs to both North and South Korea where both are allowed but also neither are allowed, where one only goes with specific purpose and explicit permission

I was a bit hesitant to do firewall rules based off of IP addresses, as a compromised host could change its IP address

Realistically any identifier you can write firewall rules based off of can be forged in some way. A rogue machine can change it’s host name, IP address and MAC address (and many do randomize their MAC address these days) in enterprises this is generally mitigated through limiting a network to only Ethernet access or via 802.1X authentication on WiFi and potentially even Ethernet. (You can also take the approach of MAC address whitelists, and some switches even allow for “sticky” MAC addresses where the first MAC address that connects is whitelisted until either the switch is rebooted or an administrator explicitly clears/allows the MAC address)

However, if each host is on its own VLAN, then I could add a firewall rule to only allow through the 1 “legitimate” IP per VLAN

You could go crazy and do everything at L3 (which your idea is basically doing but with extra steps) but that sounds like far more effort than it’s worth, since now you’re making every client also act as a router, and you lose a ton of efficiency both in configuration and in routing & switching, plus you’ve now changed the type of threats you’re vulnerable to.

Generally in the enterprise, risks like what you’re trying to mitigate are handled through reporting. An automated alert email is sent when a new device connects to a network that should never have new devices connect to it, then you kill the connection and verify with the team of that was any of them and investigate if it wasn’t.

Realistically as a home network your threat model is automated scripts and maybe a script kiddie trying to get in. You really just need higher than average security to mitigate such a threat model (and average security is a shit show)

I feel like I may have to allow a couple CT/VMs to communicate without going through the firewall simply for performance reasons. Has that ever been a concern for you?

Security is always a trade off of convenience and speed. You have to decide what is an acceptable compromise between security and efficiency

Generally anything virtual when you aren’t sure what to do, you should look at what the physical solution would be. For example, network storage is very bandwidth intensive, latency sensitive and security intensive. This is usually secured at the physical level as a separate network with no routers so that most security can be disabled. So at the virtual level these would be tackled with a separate virtual network connected to a second interface, and firewall rules on other interfaces to disallow incoming and outgoing connections to the storage network

Edit: I just realized I never answered your first question. In short, from what I’ve seen most enterprises put one firewall from a vendor like Fortinet, Zscaler, Palo Alto, etc. right on the edge of the network closest to the internet then either entirely rely on that for firewall or rely on that for firewalling off the outside Internet then do additional firewalling with a different tool inside the network. For example, a bank I worked at had a pair of redundant L3 switchs (Nexus N9ks specifically) which handled all of the routing for all of the bank’s networks, and connected between those and the internet was the Fortinet box which was managed by an outside vendor and while i was there as part of hardening ahead of a scheduled red team audit we setup firewall rules (I’m blanking on the Cisco term for it, but they’re ultimately just firewall rules) on the L3 switches to limit access to more sensitive networks and services

It really sounds like you need to dive into firewall rules. Generally you lean on your firewall to allow and restrict access to services. Probably the easiest place to start is to setup pfsense/opnsense since it has a really clean interface for setting up rules. Proxmox’s built in firewall is nice too, but configuring the firewall per VM would probably get annoying and difficult after a while

And as you learn more about firewalls learning how subnetting works will allow for more efficient rules (for example, if you have 192.168.0.0/23 192.168.2.0/24 and 192.168.3.0/ 24 for your networks that you’re allowing traffic to/from you can just enter one firewall rule for 192.168.0.0/22 rather than 3 separate rules)

I already said in the original post I plan on sellong off and giving away ~15 of them, keeping a few as spares, and only actually leaving one on 24/7

bare metal machines which take IP addresses, against just running it in VM’s which have IP addresses

Both bare metal and VMs require IPs, it’s just about what networks you toss them on. Thanks to NAT IPs are free and there’s about 18 million of them to pick from in just the private IPv4 space

Big reason for bare metal for clustering is it takes the guess work out of virtual networking since there’s physical cables to trace. I don’t have to guess if a given virtual network has an L3 device that the virtual network helpfully added or is all L2, I can see the blinky lights for an estimate as to how much activity is going on on the network, and I can physically degrade a connection if I want to simulate an unreliable connection to a remote site. I can yank the power on a physical machine to simulate a power/host failure, you have to hope the virtual host actually yanks the virtual power and doesn’t do some pre shutdown stuff before killing the VM to protect you from yourself. Sure you can ultimately do all of this virtually, but having a few physical machines in the mix takes the guesswork out of it and makes your labbing more “real world”

I also want to invest the time and money into doing some real clustering technologies kinda close to right. Ever since I ran a ceph cluster in college on DDR2 era hardware over gigabit links I’ve been curious to see what level of investment is needed to make ceph perform reasonably, and how ceph compares to say glusterFS for example. I also want to setup an OpenShift cluster to play with and that calls for about 5 4-8 core 32GB RAM machines as a minimum (which happens to be the maximum hardware config of these machines). Similar with Harvester HCI

It just takes a lot of extra power and doesn’t achieve much

I just plan on running all of them just long enough to get some benchmark porn then starting to sell them off. Most won’t even be plugged in for more than a few hours before I sell them off

there is no real reason to do this and I don’t understand so many people hyping it up.

Because it’s fun? I got 25 computers for a bit more than the price of one (based on current eBay pricing). Why not do some stupid silly stuff while I have all of them? Why have an actual reason beyond “because I can!”

25 PC’s does seem slightly overkill. I can imagine 3-5 max.

25 computers is definitely overkill, but the auction wasn’t for 6 computers it was for 25 of them. And again, I seriously expected to be out of and the winning bid to be over a grand. I didn’t expect to get 25 computers for about the price of one. But now I have them so I’m gonna play with them

I won’t be leaving all of them on for long at all. I’ve got a few basically unused 15A electrical circuits in the unfinished basement (can see the wires and visually trace the entire runs) I’ll probably only run all 25 long enough to run a linpack benchmark and maybe run some kind of AI model on the distributed compute then start getting rid of at least half of them

Although he’d also need 25 monitors lol

Back to the government auctions then!

12 cents per kilowatt-hour. I certainly don’t plan on leaving more than a couple on long term. I might get lucky with the weather and need the heating though :)

State government, and it says they come with SSDs. They came from a school so presumably they’re from a lab or are upgraded staff PCs, both would be pretty low sensitivity. Maybe I’ll learn the final test answers for Algebra 1 at worst!

Might be fun to do some forensic data recovery and see if anything was missed though

I think you’re not giving 4th gen enough credit. My wife’s soon-to-be-upgraded desktop is built on a 4th gen i5 platform, and it generally does the job to a decent level. I was rocking a 4790k and GTX970 until 2022, and my work computer in 2022 was on an even older i5-2500 (more held back by the spinning hard drive than anything. Obviously not a great job, but I found something much better in 2022) my last ewaste desktop-turned-server was powered by an i5-6500 (which is a few percentage points better performance than the 4th gen equivalent) and I have a laptop I use for web browsing and media consumption that’s got a 6700HQ in it.

I’ve already got a few people tentatively interested, and I honestly accepted the possibility of having to pay to recycle them later on. Should be a fun series of projects to be had with this pallet of not-quite-ewaste

This is pretty high on the to-do list. I plan on virtualization a bunch of it, but it would be pretty easy to have one desktop hosting each subnet of client PCs and one hosting the datacenter subnet. Having several hosts to physically network means less time spent verifying the virtual networks work as intended.

Also playing with different deployment tools is a goal too. Having 2-3 nearly-identical systems should be really useful for creating unified Windows images for deployment testing

The thought did cross my mind to run Linpack and see where I fall on the Top500 (or the Top500 of 2000 for example for a more fair comparison haha)

From the listing photos these actually have half-height expansion slots! So GPU options are practically nonexistant, but networking and storage is blown wide open for options compared to the miniPCs that are more prevalent now.

4th gen intel i5s, 8GB of RAM and 256GB SSDs, so not terrible for a basic Windows desktop even today (except of course for the fact that no supported Windows desktop operating system will officially support these system come Q4 2025)

But don’t get your hopes up, when I’ve bid on auctions like this before the lots have gone for closer to $80 per computer, so I was genuinely surprised I could win with such a low bid. Also every state has entirely different auction setups. When I’ve looked into it in the past, some just dump everything to a third party auction, some only do an in-person auction annually at a central auction house, and some have a snazzy dedicated auction site. Oh and because its the US, states do it differently from the federal government. So it might take some research and digging around to find the most convenient option for wherever you are (which could just be making a friend in an IT department somewhere that will let you dumpster dive)

This is a great writeup of errata related to this configuration! I am curious what kind of performance you’re seeing for DNS requests considering how old and anemic the first gen Pi is