I'd rule out k8s if you're looking for simple administration.
this post was submitted on 28 Jan 2024
44 points (95.8% liked)
Selfhosted
40296 readers
239 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 1 year ago
MODERATORS
The solutions you've mentioned aren't exactly equivalent. Proxmox is a hypervisor while Docker Swarm and Kubernetes are container orchestration engines. For example, I use Proxmox in a highly available cluster running on three physical nodes. Then I have various VMs and LXC containers running on those nodes. Some of those VMs are Kubernetes nodes running many Docker containers.
I highly recommend Proxmox as it makes it trivial to spin up new containers and VMs when you want to test something out. You can create and destroy VMs in an instant without messing with any of your actual hardware. That's the power of a good hypervisor.
For orchestration, I would actually recommend you just stick with Docker Compose if you want something very simple to manage. Resiliency or high-availability usually brings with it a lot of overhead (both in system resources as well as maintenance costs) which may not be worth it to you. If you want something simple, Proxmox can run VMs in a highly-available mode so you could have three Proxmox nodes and set any VMs you deem essential to be highly-available within the cluster.
For my set up, I have certain services that are duplicated between multiple Proxmox nodes and then I use failover mechanisms like floating IP addresses to automatically switch things over when a node goes down. I also run most things in Kubernetes which is deployed in a highly-available manner across multiple Proxmox nodes so that I can lose a physical node and still keep (most) of my services running. This however is overkill for most things and I really only do it because I use my homelab to learn and practice different techniques.
Definitely go with K3s instead of K8s if you want to go the Kubernetes route. K8s is a massive pain in the ass to setup. Unless you want to learn about it for work I would avoid it for homelab usage.
I currently run Docker Swarm nodes on top of LXCs in Proxmox. Pretty happy with the setup except that I can't get IPv6 to work in Docker overlay networks and the overlay network performance leaves things to be desired.
I previously used Rancher to run Kubernetes but I didn't like the complexity it adds for pretty much no benefit. I'm currently looking into switching to K3s to finally get my IPv6 stack working. I'm so used to docker-compose files that it's hard to get used to the way Kubernetes does things though.
K3s is k8s
lol at the downvote. K3s is k8s. The very first 2 words in its website are Lightweight Kubernetes. https://k3s-io.github.io/
You really should look into Nomad: https://developer.hashicorp.com/nomad/docs/nomad-vs-kubernetes
I sat up a nomad cluster in my home lab just a few days ago, on top of instances in proxmox. Works really well and is simple to maintain and understand.
First off, replace WiFi with Ethernet. Seriously, it will be way more reliable. There are plenty of janky adapters that will work fine.
Once you have that done you can setup a Proxmox cluster. Proxmox won't be a good experience with WiFi.
OP can try power line if nothing else works
Late reply but yeah, Wifi was a nightmare on Proxmox. It was a tiny e-waste SFF pc so I was able to wedge it near the other servers. The cluster is happy.
Not sure if this is what you mean, but I have my proxmox server connected via a WiFi bridge and there have been zero issues in the past year that I’m using it. Set it up to try and see if it works and haven’t had to change it so I didn’t. Mainly home assistant on the server, no media streaming or other realtime stuff, so that may be why.
I run k8s, mostly because I use it for work and really enjoy the gitops approach to management. Previously I used docker compose.
I use k8s at work a lot - I choose to use Nomad at home, you may want to add that to your shortlist.
Thank you for Nomad, will give this one a try at home!
How you liking it? Seemed a little hard to learn to me, and I do TF and Ansible.
I thought it was quite simple but it depends on your experience of course. It's a single binary and a single config file, so I felt it was soo much simpler.
You can buy a good udemy course for 10 dollars too which really helps in the beginning.
Yea it's very easy to learn enough to run, it has built-in service discovery and secrets now, and writing parameterized jobs feels so much nicer than a helm chart in k8s.
10/10, would orchestrate again
+1 for Nomad. Ive used k8s a lot and still use it, but i prefer Nomad for home purposes. You dont even need a consul cluster to run it anymore so it's pretty simple to start.
nomad + https://www.weave.works/docs/net/latest/overview/ ;) and maybe this https://github.com/ituoga/coredns-nomad
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:
| Fewer Letters | More Letters |
|---|---|
| HA | Home Assistant automation software |
| ~ | High Availability |
| IP | Internet Protocol |
| LXC | Linux Containers |
| k8s | Kubernetes container management package |
4 acronyms in this thread; the most compressed thread commented on today has 9 acronyms.
[Thread #456 for this sub, first seen 28th Jan 2024, 07:15] [FAQ] [Full list] [Contact] [Source code]
Wifi pretty much excludes k*s and I assume that swarm and Nomad would be impacted by blips in the wireless connectivity. You can try how things work out with a load balancer / reverse proxy on a wired connection, which then checks the downstream services and routes the request to available instances.
Please look into Wifi-specific issues related to the various orchestration platforms before deciding to try one out. Hypervisor is usually a win win, until you try to do failover.
Consider power line adapters instead of wifi.