Regarding different distros: yes, absolutely possible! Considering that swarm development is basically stopped the risk of inconsistencies is lower than back in the days but I’d pin the engine version on all nodes and only explicitly upgrade all at the same time to a newer version.
Regarding swarm itself, even though I understand your desire, I’d recommend to have a look at something else. Have you had a look at Nomad for instance? It’s not compose compatible but I guess these days it’s a simple prompt to convert the compose files, it’s supposed to be simpler than K8s and still offers many perks (like the failover you are looking for).
Also, from what I just read, Proxmox is working on improving the container support to allow “native” containers (inside VMs similar to Kata containers). They’re not there yet but I am wondering if it’s worth the effort from your side right now 😅
Regarding different distros: yes, absolutely possible! Considering that swarm development is basically stopped the risk of inconsistencies is lower than back in the days but I’d pin the engine version on all nodes and only explicitly upgrade all at the same time to a newer version.
Regarding swarm itself, even though I understand your desire, I’d recommend to have a look at something else. Have you had a look at Nomad for instance? It’s not compose compatible but I guess these days it’s a simple prompt to convert the compose files, it’s supposed to be simpler than K8s and still offers many perks (like the failover you are looking for).
Also, from what I just read, Proxmox is working on improving the container support to allow “native” containers (inside VMs similar to Kata containers). They’re not there yet but I am wondering if it’s worth the effort from your side right now 😅
Either way, have fun! A fellow homelab buddy 😄