Hi there, I am hosting a service for mainly me and also some family members. I want it to become more.
Currently the chain just as follows: interwebz <-> UDM <-> npm <-> services. The services are mostly run in VMs or LXCs, npm runs in docker, accompanied by one Adguard home instance and another AGH instance in an LXC.
I now want to add f2b to this whole chain. I’ve installed f2b in the OS which runs docker (and therefore npm) (VM1) and also in the OS of my existing service (VM2). Both VMs use Ubuntu as their OS.
What I already figured out thru testing and reading is, that f2b in VM2 acts as expected when the jail is triggered, and sets the block. But nftables/ufw don’t block connections coming from external, as they are coming thru npm. f2b sees and uses the public IP for the checks/bans.
So now I am thinking how to solve that.
- Keep f2b on VM2 and let it set the bans on VM1 over SSH. That’s what some people do or so I’ve read.
- Use f2b on VM1, let it check logs on VM2 and then set the bans. That would also be possible over SSH, I’d guess. But I didn’t read up on it yet.
So, what’s your opinion? How do you fine folks handle that in your environment? Option 2 seems to be more elegant, as it would be “at the entry” and let’s me easily add additional services later. But I don’t know, if that’s a path worth digging in to.
Firstly, what’s your threat model - as in what are you trying to block? For SSH access, fail2ban works. For bots and scrapers, I’m not sure how effective it is. What jails have your enabled/configured for fail2ban?
I don’t have your setup, but like someone else here said, I use Caddy on Docker as a reverse proxy for all my self-hosted services running in Docker containers. One really effective means of blocking AI scrapers and bots is Anubis - https://github.com/TecharoHQ/anubis. It might be worth a shot. It works for me.
Thanks for the suggestion, I will take a look at Anubis.
There are no SSH ports open to external, as everything I run is locally in my LAN, I do not have use for externals SSH access and don’t want to open that vector.
The services that I (want to) offer to the family, are the common selfhosters finest:
- immich
- Stirling PDF
- Jellyfin
As I do not have a authentication provider yet, the services which require an authentication by user/password are prone to brute force attacks, I want to mitigate that first and foremost.
In the mid to long run, I plan to also deploy an authentication provider for those services like Authelia, TinyAuth, PocketID, Authentik or <you name it>.
As there are so many of them and the opinion of which is best differ wildly, I have not yet decided and am in kind of a analysis paralysis. :)For auth providers, I’ve tried all the ones you named. Here’s my brief analysis:
- Authelia - deployment and configuration is moderately complex but documentation is great and it works well with Caddy reverse proxy.
- Authentik - way too complex to deploy and configure, requires you to understand their concepts very well. I’d stay away from it.
- PocketID - super easy to deploy and configure. Passkeys are easier and arguably more secure than passwords and it supports a few methods other than passkeys.
With these services you mentioned you want to or are already hosting, you could use the following in the order they are listed:
- Basic auth using Caddy (reverse proxy layer)
- Anubis to ward off bots and scrapers (between reverse proxy and hosted service)
- PocketID for auth (before service as single sign-on)
I would recommend 2 and 3 as 1 is pretty useless against a determined attacker. Anything that’s on the public internet is bound to be attacked at some point, so follow best practices and perform server hardening on the host machine. Remember, you classify the sensitivity of the data you host and determine the severity of the risk if that data were to be exposed. So work with that in mind when choosing an option.
I used fail2ban as an anti-scraper strategy for a few months. It helped with the really basic scrapers that only use one or a few IP addresses.
But then they adapted and started using hundreds or thousands of different IP addresses, each one only making a couple of requests. fail2ban became counter-productive as it slowed my server down - the log files it was monitoring became massive under high load and it needs to scan the log files often to check if an IP has been seen before.
I’m not saying don’t try it, I’m saying it’s only effective in some situations.
Thank you for bringing that up, I was wondering that too. As in: “This one has something to hide. Let’s poke harder.”
Did you try CrowdSec as someone else suggested? Could that be a mitigation as those IPs could be filtered out in the long run, if I understand the concept of CS correct?
I did not, it seemed complicated and rabbit-hole-y.
In the end I just added an Anubis-like feature into PieFed. It works great.
I have a similar setup to you, but I run caddy instead of npm. I recently added a log analyzer to caddy and noticed a bunch of unauthorized hits I thought I could do something about…
I added a region based firewall rule for the port forward address on my UniFi gateway only allowing incoming traffic from my own country. I couldn’t do a global region block as I have some legitimate incoming traffic from the USA, so I went with a rule targeting traffic to the reverse proxy ip specifically. It still managed to let in some random bot traffic, so I also added a block region rule above the allow region rule (both to the reverse proxy) blocking countries like Russia, china, North Korea etc…
If you don’t need to allow most of the world, it is an easy fix to do this at your gateway (I note you have a udm)
Thank you for suggesting the region block on Unifi, that’s a good tip.
I have that previously set in place globally under settings -> security -> protection -> region blocking.
So on a less sophisticated level and not as granular as you.
Yours seems to be much better and I will take a look at it later. At the moment I have way too many ideas and projects boiling, to add yet another one… ;)
Based on a quick read it sounds like the issue will be that the bad actors won’t get dropped immediately. Meaning whatever servers and services are in front of the firewall that fail2ban configures, will still receive the performance hits. But your backend services will still be protected.
I have a server specifically set up as the ingress server. That runs fail2ban and Traefik and I have crowdsec set up with a Traefik bouncer plugin. So fail2ban runs in front of everything to catch the worst, most obviously bad stuff and drop the traffic very quickly and efficiently so it doesn’t get any further including things trying to access services other than the web services. Then crowdsec is focused on more complex threats as they flow through the reverse proxy. Seems to work the most efficiently for me, but I haven’t had a chance tondo any real analysis on how much improvement it might be over any other configuration.
Thanks for bringing it up.
Before I dove into that project, I was checking logs on npm and the service I am already exposing. It seems to me, both only get http GET requests which on npm all get a 404 response.
I don’t remember what the service does with those, but I will investigate later. As I didn’t panicked, I guess it wasn’t that bad.There are basically no brute force attacks on the service yet and the admin account is only allowed to login from internal IPs.
One initial aspect to further harden the exposed systems were the logs in the UDM which show frequent blocked attempts of varying severity.
This may be security theater and I am not able to assess whether the UDM is a “good enough” protection of the most common attacks.As I see it, at the moment the UDM already blocks most of the (more sophisticated) attacks and leaves only common requests for the services to handle.
Long story short, as I see it, the UDM has the role of ingress server in my setup.
You have already found the core issue: a ban on VM2 is too late.
VM2 does see the real client address in the HTTP logs if NPM passes
X-Forwarded-Forcorrectly, but at the network layer every connection to VM2 still originates from NPM on VM1. Therefore an nftables/UFW rule on VM2 can only block NPM — which is not exactly the intended security feature.For your setup, I would put the actual enforcement at the ingress point:
- Run Fail2Ban on VM1.
- Parse the NPM access/error logs there.
- Let Fail2Ban add/remove bans locally on VM1.
- Ensure NPM logs the real client IP, and only trust forwarded-IP headers from proxies you actually control.
- Keep VM2 restricted so it accepts service traffic only from VM1/NPM where possible.
Having Fail2Ban on VM2 execute remote firewall actions on VM1 via SSH can work, but it is basically building a small, brittle distributed ban system yourself: SSH keys, narrowly scoped sudo rules, reliable unban actions, error handling, and so on.
Alternative:
If you want detection from several VMs/services but enforcement centrally at NPM, CrowdSec is a more natural fit. Run agents where the relevant logs live, use a central LAPI, and run a bouncer at VM1/NPM. Then the components are designed to exchange decisions instead of hoping two independent Fail2Ban installations telepathically coordinate.Also: do not use
set_real_ip_from 0.0.0.0/0just to makeX-Forwarded-Forwork. That turns a client-supplied header into an IP-spoofing API.#SelfHosting #ReverseProxy #Fail2Ban #CrowdSec #NginxProxyManager
Thank you very much for your input. I will look into the third option.
Thought of it, but did forget it over tinkering the last few days. :)You’re welcome — happy tinkering. 🙂
