My homelab is essentially my own passion project and only really I access it except for when I spin up the occasional game server for friends.

I’m currently running Proxmox and run a debian LXC container for each docker stack I have, and have OpnSense routing incoming traffic with Haproxy with ssl offloading. My currently running LXCs are: mediawiki, amp game server(2 Minecraft servers), freshrss, and currently playing around with n8n.

I’m looking to collapse my LXC’s to just VMs. I’d like to be able to have 3 VMs running in a Docker Swarm together so I can upgrade a VM at a time and just swing my running containers to another docker node and then swing back when the VM is stable again.

I’ve looked at k0s, k3s, and k8s and it just seems way too much work and overhead for what I’m willing to do. I also want to keep using docker compose and want a decent webgui to manage my containers/nodes/swarm. I’m using DockHand right now, but need to research swarm support.

Anyone have any advice for something like this? Any specific terms, tech, software I should look into?

Also, gonna throw a curveball, but what would the effects be of running 3 different distros as my nodes in my swarm? Like a Debian node, Rocky Linux node and potentially arch node? I’m guessing I shouldn’t due to docker engine differences potentially.

I’m just trying to have fun with things, break things, fix them, learn, etc.

  • melfie@lemmy.zipEnglish
    2·
    2 hours ago

    I run k3s on a single node and it’s not really that much more overhead than Docker Compose if you understand k8s. I mostly have a deployment.yaml, service.yaml, ingress.yaml, and network-policy.yaml for each service that I’ve copy / pasted and updated. Here are some of the benefits over Docker Compose for my setup:

    • Has a built-in Traefik reverse proxy / ingress controller so I can access my services by domain name instead of by port, like http://jellyfin.lan/, http://forgejo.lan/ (using local dns on my OpenWRT router)

    • I use the Calico CNI so I can have network policies for each service to allow them to access only what they need. If a service doesn’t need internet access, it doesn’t get it.

    • I use Bitnami Sealed Secrets to store my secrets in YAML files that can be safely stored in git

    • ConfigMaps make it easy to manage configuration files

    • Easier to have separate YAML files for each service while sharing a network between them. Services connect to each other like http://forgejo.forgejo.svc.cluster.local/

    Of course, if you’re looking to load balance across multiple machines, k3s makes even more sense.

    Edit:

    k8s is the clear industry standard for container orchestration at this point, so if you want something beyond Compose, a lightweight k8s distribution like k3s is an obvious choice.