• 0 Posts
  • 169 Comments
Joined 3 years ago
cake
Cake day: July 2nd, 2023

help-circle
  • The filesystem is a database as well.

    Only in the most reductionist sense would this be true. In practice, a relational database (ie has a schema and querying language) is no replacement for a hierarchical filesystem, and vice versa.

    A filesystem stores binary blobs of data, and its up to each file format to define the semantics of the data contained. A database is a data structure coupled with accessors to present and cross-reference structured data. Trying to do a LEFT-JOIN on three binary files is a category error. And storing a PNG in a MariaDB table would be a sort of malpractice.

    A closer comparison would be an SQL database versus a data serialization format like YAML. Both have structure and have data that can be acted upon in specific ways. Arithmetic can be performed on numeric values, and strings can be concatenated together. But neither will let you concatenate numeric values, such as 2 + 2 = “22”. You need a programming language like PHP to perform such shenanigans.

    In the context of serving web content, the filesystem is the domain of the OS, meaning it has been honed by decades of experience to make it as performant as possible, built into the kernel to utilizing whatever caching tricks that make sense, and this is available irrespective of the specific userspace stack (eg LAMP) that is running. However, no mainline OS has a relational database built into the kernel and is available for a web application to use. Database engineers go through great pains to optimize a system to run as a database server.

    Phrased another way, there are no mainline OS’s that omit the filesystem. So removing the need for a relational database is removing an attack surface, removing a dependency, removing another thing that can break. The point of a database is to look up pieces of data. But if a web server can just serve up a whole HTML file that includes all the data needed, then the database can be omitted and performance will be higher as a result.

    Can a filesystem be used in lieu of a database? Sure but there are many things which can be done but shouldn’t in all normal circumstances. That is the crux of engineering: to select the right tool for the job.


  • If scale is a concern, then you want to be reducing drag as much as possible. Database lookups and edits can get expensive, so maybe just avoid them outright: DokuWiki only requires a web server with PHP. It keeps content as plain files, making it easy to serve up and to back up.

    But it’s often said that perfection is when nothing can be taken away, so what if we remove PHP as well? In this case, we need the content to already be rendered HTML. And we can do that, since there’s a body of site generation packages that build from Markdown files. I found this one while randomly searching: https://codeberg.org/milofultz/swiki

    At this point, it’s just a plain web server that dishes out static HTML files. Such simplicity will withstand most AI scrapers, because the cost per request is now absurdly low. And caching a static site is not particularly difficult. Indeed, if the sum total of the wiki content is small enough, it might even fit into something like Codeberg Pages, which means we’ve eliminated even the web server (though this would depart from c/selfhosted).

    What seems to be missing from that swiki package – but is entirely feasible – would be to have all content in Markdown files and live in a Git repo, and the Git history itself is used as the Wiki history. This means your pages would retain the classic wiki change log, so that it can all be generated from a Git repo that’s small enough to keep on a floppy disk.

    Simplify, then add lightness.


  • This and the comment it replies to make the categorical error which is to assume that NAT exists as part-and-parcel to firewalling. This is not the case. Many American university and corporate networks do still adhere to the end-to-end principle by giving every device a public IP address. They can do this because they are among the few legacy entities that actually have enough for their needs, such as a Class A (aka /8) subnet on Legacy IP. At the same time, these entities employ firewalls that don’t allow unsolicited inbound traffic (and sometimes also block disallowed outbound traffic).

    NAT is not a firewall, and the OP correctly points out that by using NAT or CGNAT, the end to end principle is demolished, with no way for the victims to recover. OP mentions “firewalls” exactly twice, to compare how they are not firewalls and are still necessary in spite of NAT. Nobody is credibly suggesting that everyone should switch to IPv6 without a firewall. Every single domestic router for the past two decades has included a firewall, and that would not change in an all IPv6 world.

    I wish to reiterate an example I’ve used in the past, which is that the most famous address in the USA is 1600 Pennsylvania Ave, Washington DC. Anyone can send letters and parcels directly to this address as-written, with no need to rewrite the address to first arrive at the Secret Service. And yet, all such parcels and letters will be thoroughly scrutinized by the Secret Service anyway at an off-site facility, and some indeed will be let through to the occupants of the residence. NAT is address mutilation, and the United States Postal Service does not require nor allow that.


  • What are these “built in listeners” that you speak of, and which OS’s can I find this on?

    Also TCP will work before UDP. For whatever reason firewalls and such do weird things with UDP traffic.

    This doesn’t really explain why troubleshooting is more difficult, but rather that firewalls often cause trouble for UDP. My question was “why would diagnosing a UDP problem be harder than anything else?”, not “why is UDP more problematic?”. I’m fully aware that some firewalls are configured with absurd parameters, like blocking ICMP, QUIC, or even TLS in some bizarre situations. But my tools for debugging network troubles have always been the same: tcpdump for passive tracing, nc to coerce an active response, and icmp/icmp6 ping for basic connectivity and MTU verification.

    Whether it’s UDP or anything other protocol, those tools have proven sufficient for me. Do you use something else?


  • I’ll add some color to this post.

    The original FTP is, frankly, a monster of a protocol. Very useful, but an anomalous protocol that even without NAT breaking the end-to-end principle, it is unlikely to have survived modern corporate firewall rulesets in any case. The fact that FTP was even ported for TCP is its own historical quirk.

    BTW, that is the term which this post is missing: the end-to-end principle is the design philosophy that the network itself should not have to perform work on payloads transmitted, except to carry it towards the destination. This also implies that no fields or bits should be modified in transit, once it leaves the sender; encap/decap restores the fields so that the receiver is none the wiser.

    NAT breaks end-to-end in two respects: corruption of the original sender, and corruption of L4 port numbers. Note that Legacy IP also violated end-to-end, when packets are fragmented due to MTU issues. With IPv6, fragmentation by the network is disallowed outright, and the technical case for NAT is non-existent.

    There’s something to be said about adopters of NAT, that they were facing a Hobson’s choice: use NAT so that they could connect early dial-up users to the emerging Internet, or reinvent the 20 year experiment of the Internet so that it could scale properly. In 1995, IPv6 did not exist so they made the only real choice available.

    In the year 2026 though, that argument doesn’t hold water: it is a choice to continue to ignore the dual-stack and IPv6-only internet. There’s a quote that all sufficiently complex technical problems are in-fact political, and this is that: the modern case for Legacy IP and NAT is rooted in inertia, resource scarcity (eg hyperscale cloud companies buying up IP subnets), and recalcitrance by professionals that abdicate their responsibility to their clients to pursue the available technology.

    There is no colorable technical rationale for why IPv6 best practices cannot be adopted today for most organizations, when all network hardware, all major consumer and enterprise OS’s, and all mobile phones support v6. Note that I said “best practices”, because a minority of orgs such as certain American ISPs have undertaken truly bonkers decisions that are putting us on track for the very same sins as Legacy IP.

    The fact that some ISPs assign nothing but a single /128 via DHCP6 is absurd: this malpractice perpetuates the same problem as NAT44, except that there’s no good excuse for it. Even the most delusional of ISPs will never run out of /128 addresses in their assigned /32. Per best practice, even handing out /48’s to customers is not a problem either, because that’s 65536 customers and if that’s really a problem, just ask the RIR for another /32, which they can do as RIR dues-paying members. There is no practical limit, except that some people just cannot math properly to see that there’s no practical limit. When a technical solution to a technical problem fails because of innumeracy by those tasked with implementing it, then that’s so much worse than any 1990s workaround.

    I’ve harped a lot about IPv6 because its strength today is that it’s technically competent, future-proofed, and most importantly, practitioners that saw the first travesty of NAT will not allow a redux to play out with IPv6. The technical reasons to deploy NAT66 are non-existent: it is always a workaround for political issues.

    I can (just barely) accept NPTv6 as a like subnet-for-subnet mapping that does not harm L4, but that’s still glossing over a political issue. IPv6 gave network engineers real choices in numbers and administering their networks, and while some will squander it, I will be encouraging people to not let that happen.


  • Why would UDP be any more difficult than TCP (or anything else) for troubleshooting? If nothing else, it should be easier because there are no stateful connections and every packet is “fire and forget”. Now, as a tunneling technology, WG might be more difficult than a simple client/server socket-driven application, but that just means some routing tables need to be inspected and you have to understand IP subnets. In any case, you’d still want to obtain packet traces from both ends of the WG tunnel.


  • The thing is, if you don’t wish to port-forward on Legacy IP nor open firewall ports for IPv6, then the only other option is to reach out to some sort of relay server. But whether your own VPS fulfils that role or you use Tailscale’s DERP/TURN relays, the same class of latency and bandwidth issues will still erupt. So you’d still have to debug them, although you presumably would have better visibility into diagnosing issues with your VPS relay.

    Generally speaking, if Tailscale isn’t working for you, then all other solutions will increase in complexity, not decrease.

    Answering the ultimate question, I operate and secure my SSH server, with port 22 open only for IPv6 inbound. There’s no NAT, so I don’t have to deal with debugging odd port mangling. The SSH server only allows key auth, and all password attempts are logged and blocked, because there can’t be any legitimate traffic like that. On this SSH server would be my mounts for backups.







  • In the space of mini PCs/NUCs, this does seem comparable. But bear in mind that the entire space of “home server” is a lot bigger than NUCs. After all, not everyone just wants to host an NVMe drive or two; they might want to fit some cheaper SATA SSDs or even spinning rust, as part of a cascade of stroage, with caching on SSDs and other nice things.

    Likewise, the dual LAN ports with 802.3bz support is nice, but the reality is that most networks are only 1 Gbps with some devices supporting 2.5 Gbps. Quite frankly, for server use that actually needs one or both port’s bandwidth, the link would likely be better served with SFP+ slots, because of better compatibility with cheap enterprise switches available online, and because SFP+ transceivers run less hot than their twisted pair counterparts at 10 Gbps. Heck, you could even use a 40 Gbps QSFP+ NIC, which only requires a x8 or X4 PCIe slot (depending on PCIe generation of the NIC). On that note, a single PCIe slot is nice, but it does preclude the use of a double-wide GPU for transcoding use.

    I would sum all this up as lacking expandability. Yes, it can be expanded somewhat. But for folks that want the capacity of an ATX-based build, this ain’t it. And for folks with abundant solar power, electricity consumption for older gen chips isn’t as much of a dealbreaker. It certainly checks off a lot of people’s boxes, yours included, but that doesn’t mean that people are “sleeping on” this, because at the end of the day, if it doesn’t serve their purposes, it’s not the right box for them. In terms of price, never underestimate the cost of $0.00 that is an old, unused laptop; they make remarkably decent servers for light loads.

    Also, as of this writing, only the 32 GB RAM variant is available on the USA website.


  • Would cert-manager for Kubernetes be approximately what you’re looking for? In that particular case, it’s a service that runs in a K8s cluster, that can auto-request CSRs and send them to Lets Encrypt using one of the challenge types. It takes care of managing the returned certificates, such as renewing them before expiration and registering the private keys within the K8s key store.

    I presume that you’re not already using K8s, so maybe that project will help you find the keywords to describe the sort of certificate manager that you want.


  • If you’re already using Debian on desktop, then Debian on a VPS makes sense from a maintenance perspective: you only have to track one set of security updates. When there’s suddenly a critical update for some vulnerability, you can update both at once because it’s the same upstream repos. Whereas with Ubuntu on your VPS, you’d have to be mindful to make sure to check back when the same vulnerability is patched there.

    Do not underestimate the effort to upkeeping a server, and do try to unify your procedures so you’re spending less time doing the necessary things, and more time building up your self hosting stack.


  • I’ll offer a comparison not yet mentioned in the context of hosting software, but which should sound familiar in the social media context: moderation is hard at scale.

    By every metric of scale, Codeberg is a competitor to the behemoth that is GitHub. And that means they have to contend with tough choices, such as who and what to host on their servers, no different than BlueSky, Facebook, and the individual instances of the Fediverse. And to be clear, Codeberg has always been limited to FOSS projects anyway, so nobody can genuinely argue that limits weren’t in the cards.

    The distinction is that less alike Meta, and more alike with the Fediverse, Codeberg takes no hostages: people with obvious AI repos (or even marginal repos, given the possible ambiguity) are free to take their code and leave. Yes, they have to rebuild their CI pipelines, but that’s always required for any platform, including self hosting.

    The fact is that Codeberg is a centralized entity that needs to (try to) make consistent policy. In this case, they have a concern about copyright and nothing anyone says can magically dematerialize that risk. If their new policy is not agreeable, then decentralized self hosting is probably the only way forward. But that was always an option with ForgeJo and federation.

    And indeed, folks moving from GitHub or elsewhere are not exactly without fair warning that they might have to migrate again; we’ve not reached a utopia yet.


  • What is the absolute simplest way to accomplish this?

    < gets on soap box>

    The absolute simplest way is to use IPv6: on your Debian machine, assign multiple IPv6 addresses, one for each service you want to expose to your LAN. There’s no penalty with having a dozen v6 addresses.

    Although you have those addresses, your machine generally uses just a single one for its own outbound and inbound traffic. For each service, you would edit their config so that they bind to a specific IPv6 address. Finally, you would configure an AAAA entry in DNS so that your chosen subdomain will point to the IPv6 address in question.

    As an example, suppose you had three different web servers running Nginx, currently on port 80, 8081, and 42069. What you would change is the server config for each server instance, adidng the IPv6 address as part of the “listen” directive. Since each instance is now bound to a different IP address, nginx can now listen to the conventional port 80 and n’ary will the three collide. In other software, the configuration option you’re looking for is the “bind address”. By binding each app to its own IP, it will only respond if you send a request to that IP (or DNS name, which translates to an IP); this conveniently makes debugging really easy.

    For a home network, you might have a single /64 IPv6 subnet. But that still means you literally have billions upon billions of addresses to use before you ever run out. And as you’ve recognized, using DNS names to identify services is a lot easier and intuitive than using port numbers. It also neatly avoids the need to memorize IPv6 addresses, because that’s never been necessary: we have DNS.

    If your ISP won’t give you IPv6, you can still use locally-assigned private ranges, known as ULA, and this works because your services are contained to your LAN. Best practice is to randomly generate a subnet then use it.

    < gets off soap box>


  • litchralee@sh.itjust.workstoSelfhosted@lemmy.worldImmich vs Ente ?
    link
    fedilink
    English
    arrow-up
    7
    ·
    edit-2
    2 months ago

    Is that something you’re looking for? As a practical matter, the effort to maintain multiple release branches and backports – compared to the norm of just maintaining a single main branch – is an outsized effort, available to only well-funded FOSS projects, usually by having an industry sponsor that makes it a priority.

    I posit that the grand majority of selfhosted FOSS software, by project count, do not have back ported branches. And so production environments will want to set up an offline validation setup to evaluate update, prior to making the switch. Not convenient, sure, but not exactly insurmountable either.

    At some point, some amount of responsibility for use of FOSS software must fall upon the users, or else the project is less about creating value by building up the commons for software, and more on the exploitation/abuse of the volunteers.