Holy setup batman. Was thinking it was going to be another container I spin up, but it’s enabling kernel modules, needs IOMMU, needs a ton of setup and then it looks like you still have to compile it? For now at least that’s above my needs

but the vast majority of crawlers don’t care to do that. That’s a very specific implementation for this one problem. I actually did work at a big scraping farm, and if they encounter something like this,they just give up. It’s not worth it to them. That’s where the “worthiness” check is, you didn’t bother to do anything to gain access.

That’s counting on one machine using the same cookie session continuously, or they code up a way to share the tokens across machines. That’s now how the bot farms work

I was a single server with only me and 2 others or so, and then saw that I had thousands of requests per minutes at times! Absolutely nuts! My cloud bill was way higher. Adding anubis and it dropped down to just our requests, and bills dropped too. Very very strong proponent now.

This dance to get access is just a minor annoyance for me, but I question how it proves I’m not a bot. These steps can be trivially and cheaply automated.

I don’t think the author understands the point of Anubis. The point isn’t to block bots completely from your site, bots can still get in. The point is to put up a problem at the door to the site. This problem, as the author states, is relatively trivial for the average device to solve, it’s meant to be solved by a phone or any consumer device.

The actual protection mechanism is scale, the scale of this solving solution is costly. Bot farms aren’t one single host or machine, they’re thousands, tens of thousands of VMs running in clusters constantly trying to scrape sites. So to them, a calculating something that trivial is simple once, very very costly at scale. Say calculating the hash once takes about 5 seconds. Easy for a phone. Let’s say that’s 1000 scrapes of your site, that’s now 5000 seconds to scrape, roughly an hour and a half. Now we’re talking about real dollars and cents lost. Scraping does have a cost, and having worked at a company that does professionally scrape content they know this. Most companies will back off after trying to load a page that takes too long, or is too intensive - and that is why we see the dropoff in bot attacks. It’s that it’s not worth it for them to scrape the site anymore.

So for Anubis they’re “judging your value” by saying “Are you willing to put your money where your mouth is to access this site?” For consumer it’s a fraction of a fraction of a penny in electricity spent for that one page load, barely noticeable. For large bot farms it’s real dollars wasted on my little lemmy instance/blog, and thankfully they’ve stopped caring.

Check out Anubis. If you have a reverse proxy it is very easy to add, and for the bots stopped spamming after I added it to mine

This looks great! Thank you for the recommendation!

Interesting, had no idea!

My mind went to this one

It’s out of date, and in desperate need of a rewrite. PHP might have been an okay choice 15 years ago, but no one in their right mind should be using PHP for modern server development. (Yes I’m calling out Pixelfed too). With so many languages and frameworks, that’s probably one of the worst right now.

Then it was proven that they don’t really get modern infrastructure either, as their docker containers depend on stateful code, with combinations of environment variables and php files that need to be stored in volumes, and then plugins which are also stateful - meaning that on new updates they need to go through an “update” process. This is directly opposite of good practice as docker containers should be 100% immutable and be able to run just by using docker run. They also have required volume mounts scattered throughout the OS, it was just never designed with containers in mind.

I can’t recommend nextcloud right now, it’s incredibly brittle and slow.

Agree with others, if you try to do a replica it’s going to be very inefficient, and your costs will be high. You’re looking for a backup, then just nightly/weekly you perform your backups. Any blob storage then will do, just work out what pricing works for you. Just plan out how you’d do a restore in case everything came crashing down - from ground up how would you bring your services back online?

HDD enclosure is a fine way to start, as long as you know it has limitations. Eventually you’ll probably need more storage, and it won’t scale. That being said, you can get 26TB hard drives now, it’ll be a while. Just make sure you plan out how to back it up. Remember the rule - if you can’t afford to buy a backup then you can’t afford to do the project. Make sure you have backups in mind.

If you decide to upgrade to a full NAS solution later also remember that during that migration you probably will need to use new hard drives while migrating as your current ones will need to be copied from to the new NAS, meaning you will probably end up with a few redundant drives. Not a huge thing, but there will be no “in-place” upgrade. It all depends on where you want your homelab to go in the future.

Yeah Proxmox leaves a lot to be desired in terms of metrics. However, metrics are supported out of the box. Bad news, you probably won’t get what you want within proxmox. Good news, you have another project you get to undertake! Hooray!

Like I said, Proxmox supports metrics out of the box. If you go Cluster -> Metric Server you’ll be able to see that you can add a metric server. The first iteration I did with proxmox I added an InfluxDB container which then proxmox can talk to (yes they can be on the same host), and then proxmox will start pumping metrics into InfluxDB. (It uses Telegraf under the hood). Then, you can also run Grafana, add your InfluxDB as a data source, and then you have a sweet metrics dashboard. There are a lot of pre-built dashboards already made that look great, and you can customize from there.

You can also use Graphite, I personally haven’t used it, but I also dropped Influx over time too. These things evolve in that sort of way. That’s how I’d get set up and started though.

It depends on your setup of course, many ways to skin that cat. You didn’t really say where you wanted dashboarding. Do you want it through a terminal? Through a UI within proxmox? Personally I took it as an opportunity to learn Grafana and hosted it

Did a takehome for a company recently that did it well. They required that I make a docker file (you could give them one if you wanted) where when ran it would run tests. It was a neat use of docker IMO, it standardized that builds were just “build the docker file” and running was just “run the dockerfile”. You would t have to deal with tar or anything then.

Thousand ways to skin a cat there

deleted by creator

They said they were open to it but they had zero priority of doing it themselves, and essentially “submit a PR if you want it”. A shame really, their interface is great, and such an easy setup. If they implemented either xmpp or matrix I would switch immediately. All of my friends want a discord clone that “just” works, but no one wants to go to this server for this group and then login to that server for that group. They want a single-pane interface like what discord offers.

Shortsighted to not implement that IMO.

Yeah I would assume if anything they would have helped them bring federation

Yeah yeah yeah but ignoring that… Would you?

You can get away with a pretty low bitrate for most. As others have said, set the egress upload limit in the app to whatever you prefer, and just be ready to transcode.

Never charge for access to your server in any way. That is officially 100% illegal. If you can’t do it without charging, you can’t do it