this post was submitted on 20 Feb 2025
37 points (95.1% liked)

Selfhosted

42676 readers
621 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
ruud@lemmy.world
Loki@lemmy.world
CannaVet@lemmy.world
devve@lemmy.world
HybridSarcasm@lemmy.world
HybridSarcasm@lemmy.hybridsarcasm.xyz
37
Docker in LXC vs VM (lemmy.ml)
submitted 2 days ago by mr_jaaay@lemmy.ml to c/selfhosted@lemmy.world
53 comments fedilink hide all child comments
 

I run a small server with Proxmox, and I'm wondering what are your opinions on running Docker in separate LXC containers vs. running a specific VM for all Docker containers?

I started with LXC containers because I was more familiar with installing services the classic Linux way. I later added a VM specifically for running Docker containers. I'm thinking if I should continue this strategy and just add some more resources to the docker VM.

On one hand, backups seem to be easier with individual LXCs (I've had situations where I tried to update a Docker container but the new container broke the existing configuration and found it easiest just to restore the entire VM from backup). On the otherhand, it seems like more overhead to install Docker in each individual LXC.

top 50 comments
sorted by: hot top controversial new old
[–] MangoPenguin@lemmy.blahaj.zone 10 points 2 days ago* (last edited 2 days ago) (2 children)

Regardless of VM or LXC, I would only install docker once. There's generally no need to create multiple docker VMs/LXCs on the same host. Unless you have a specific reason; like isolating outside traffic by creating a docker setup for only public services.

Backups are the same with VM or LXC on Proxmox.

The main advantages of LXC that I can think of:

  • Slightly less resource overhead, but not much (debian minimal or alpine VM is pretty lightweight already).
  • Ability to pass-through directories from the host.
  • Ability to pass-through hardware acceleration from a GPU, without passing through the entire GPU.
  • Ability to change CPU cores or RAM while it's running.
[–] possiblylinux127@lemmy.zip 2 points 1 day ago

Also LXC shares the host filesystem to there is less concern with corruption due to power loss.

[–] non_burglar@lemmy.world 1 points 1 day ago* (last edited 1 day ago) (2 children)

I use individual lxc for each docker compose so I don't have to revert 8 services at once if I need to restore.

I would also argue that an alpine lxc runs in 22mb ram by itself ... Significantly smaller footprint on disk and in memory. But most importantly, lxc can actually share memory space effectively, one doesn't need to reserve blocks of ram.

[–] sugar_in_your_tea@sh.itjust.works 5 points 1 day ago* (last edited 1 day ago) (1 children)

You don't have to revert 8 services, you can stop/start them independently: docker compose stop <service name>.

This is actually how I update my services, I just stop the ones I want to update, pull, and restart them. I do them one or two at a time, mostly to mitigate issues. The same is true for pulling down new versions, my process is:

  1. edit the docker-compose file to update the image version(s) (e.g. from 1.0 -> 1.1, or 1.1 -> 2.0); I check changelog/release notes to check for any manual upgrade notices
  2. pull new images (doesn't impact running services)
  3. docker compose up -d brings up any stopped services using new image(s)
  4. test
  5. go back to 1 until all services are done

I do this whenever I remember, and it works pretty well.

[–] non_burglar@lemmy.world 2 points 1 day ago

Oh, nice. Thanks!

This is me showing my docker ignorance, I suppose.

[–] MangoPenguin@lemmy.blahaj.zone 2 points 1 day ago (1 children)

How do you handle backups? Install restic or whatever in every container and set it up? What about updates for the OS and docker images, watchtower on them I imagine?

It sounds like a ton of admin overhead for no real benefit to me.

[–] non_burglar@lemmy.world 1 points 1 day ago

I just snapshot the parent lxc. The data itself isn't part of the container at any level, so if I bung up compose yml or env, I can just flip it back. The only real benefit is that all my backups are in the same place in the same format.

Like I'm not actually opposed to managing docker in one unit, I just haven't got there yet and this has worked so far.

If I were to move to a single platform for several docker, what would you suggest? For admin and backups?

[–] possiblylinux127@lemmy.zip 5 points 1 day ago

Honestly you can do either.

LXC

  • shares host kernel (theoretically lighter weight)

  • less isolation from host (less secure)

  • devices are passed via device files

  • less flexible due to dependence on host

  • no live transfers

  • filesystem shared with host

virtualization

  • has own kernel and filesystem

  • supports live transfers

  • hardware pass though is done at the device level

  • more flexible due to independent kernel

  • more overhead

[–] bizdelnick@lemmy.ml 9 points 2 days ago (3 children)

What's the purpose of running container in a container? Why not install docker on your host machine?

[–] DarkDarkHouse@lemmy.sdf.org 8 points 2 days ago (2 children)

If you do that, Docker is stuck on that host. If it’s in an LXC it can move to another host. Plus, backing up and snapshotting are easier IMO.

[–] bizdelnick@lemmy.ml 5 points 2 days ago

Snapshotting in docker is as easy as docker commit. After that you can back it up with docker save. Then move to another host, but not without downtime.

However normally you need to backup/move only volumes attached to containers. If that's not the way how you like to organize your services, you likely don't need docker.

[–] just_another_person@lemmy.world 4 points 1 day ago

Docker doesn't need to portable because containers are...

I don't even understand this logic.

[–] MangoPenguin@lemmy.blahaj.zone 2 points 2 days ago (2 children)

Dockers 'take-over-system' style of network management will interfere with proxmox networking.

[–] bizdelnick@lemmy.ml 1 points 1 day ago

Well, I don't use proxmox, however docker coexists with libvirt and other virtualization systems. If there are overlapping networks that docker ant proxmox attempt to manage, they are configurable.

[–] sugar_in_your_tea@sh.itjust.works 1 points 1 day ago* (last edited 1 day ago) (1 children)

I don't use proxmox, but it works absolutely fine for me on my regular Linux system, which has a firewall, some background services, etc. Could you be more specific on the issues you're running into?

Also, I only really expose two services on my host:

  • Caddy - handles all TLS and proxies to all other services in the internal docker network
  • Jellyfin - my crappy smart TV doesn't seem to be able to handle Jellyfin + TLS for some reason, it causes the app to lock up

Everything else just connects through an internal-only docker network.

If you're getting conflicts, I'm guessing you've configured things oddly, because by default, docker creates its own virtual interface to explicitly not interfere with anything else on the host.

[–] MangoPenguin@lemmy.blahaj.zone 3 points 1 day ago (1 children)

A couple posts down explains it, docker completely steamrolls networking when you install it. https://forum.proxmox.com/threads/running-docker-on-the-proxmox-host-not-in-vm-ct.147580/

The other reason is if it's on the host you can't back it up using proxmox backup server with the rest of the VMs/CTs

[–] sugar_in_your_tea@sh.itjust.works 2 points 1 day ago (1 children)

I don't use proxmox, so I guess I don't understand the appeal. I don't see any reason to backup a container or a VM, I just backup configs and data. Backing up a VM makes sense if you have a bunch of customizations, but that's pretty much the entire point of docker, you quarantine your customizations to your configs so it's completely reproducible if you have the configs and data.

[–] MangoPenguin@lemmy.blahaj.zone 2 points 1 day ago* (last edited 1 day ago) (1 children)

Ease of use mostly, one click to restore everything including the OS is nice. Can also easily move them to other hosts for HA or maintenance.

Not everything runs in docker too, so it's extra useful for those VMs.

[–] sugar_in_your_tea@sh.itjust.works 1 points 1 day ago

That's fair.

That said, I can't think of anything I'd want to run that doesn't work in docker, except maybe pf? But I'd probably put that on a dedicated machine anyway. Pretty much everything else runs on Linux or has a completely viable Linux alternative, so I could easily built a docker image for it.

[–] mr_jaaay@lemmy.ml 1 points 1 day ago

Honestly, I never really thought of installing Docker directly on Proxmox. I guess that might be a simpler solution, to run Dockers directly, but I kind of like to keep the hypervisor more stripped down.

[–] Pulsar@lemmy.world 6 points 2 days ago

I have been run Docker container in both LXC and VM for a long time without issues or meaningful performance penalties. So I run important single docker containers on top of LXC and everything else in Dockge / Portainer VMs.

[–] tofuwabohu@slrpnk.net 5 points 2 days ago

I can't say much to docker in LXC as I'm not using it, I vaguely remember some limitation I've read of but if it works fine for you those don't seem to apply.

A VM has more overhead than an LXC, but with several LXCs maybe a single VM wins on overhead.

I currently have most Docker containers in one VM and am thinking about splitting it, the main reason is that 2 deployments have way larger volumes than the rest. This leads to the snapshots of the VM being very large as well and if I would need to restore from snapshots for a "small" application, it would take super long because of the large ones.

A single VM may be a bit easier on maintenance than several LXCs.

If you don't have a specific reason to switch, I would not.

[–] DarkDarkHouse@lemmy.sdf.org 4 points 2 days ago

You can also create a single LXC for Docker and run multiple Docker containers on it. The VM argument is for security as it’s harder to escalate to the host from a VM than from an LXC.

[–] Nephalis@discuss.tchncs.de 3 points 2 days ago

You could create a fresh container, install docker, and create a new template image from it. This way the overhead of installing disapears. The overhead in resource usage for each docker installation would remain the same as before.

As mentioned in another reply, you could run several container in one lxc. For example with docker compose or podman. Since I have no experience with podman but with docker compose, docker compose is pretty simple.

But all in all, I prefer to install everything "bare metal" in lxc containers. The main reason is, I don't want to mess around with the extra layer of configurating ports etc.

[–] conrad82@lemmy.world 2 points 1 day ago (3 children)

I used to use LXC, and switched to VM since internet said it was better.

I kinda miss the LXC setup. Day to day I don't notice any difference, but increasing storage space in VM was a small pain compared to LXC. In VM I increased disk size through proxmox, but then I had to increase the partition inside VM.

In LXC you can just increase disk size and it immediately is available to the containers

[–] hendrik@palaver.p3x.de 3 points 1 day ago

I don't think the internet gave particularly good advice here. Sure, there are use-cases for both, and that's why we have both approaches available. But you can't say VMs are better than containers. They're a different thing. They might even be worse in your case. But I mean in the end, all "simple thruths" are wrong.

[–] Oisteink@feddit.nl 3 points 1 day ago

Dont listen to them! The main issue with containers vs vm is security as you lxc runs in the hosts, while a vm runs on the host.

Use what you are familiar with and remember that lxc are containers and docker are containers, but the use of them are vastly different.

[–] possiblylinux127@lemmy.zip 1 points 1 day ago (1 children)

Personally I just Mount file shares within the VM

[–] conrad82@lemmy.world 1 points 1 day ago (1 children)

I tried that too for a time, using samba. But databases didn't work from a share. I just found it easier in the end to have volumes inside the LXC / VM directly

[–] possiblylinux127@lemmy.zip 2 points 1 day ago (1 children)

Using Samba for a database is crazy. You want unencrypted NFS.

Databases aren't all that big in my case so I usually just leave them be.

[–] conrad82@lemmy.world 1 points 1 day ago

When I have used nfs in the past, i have issues with different user ID. What is the best solution these days?

After becoming a father last year, the time I have for tinkering is close to 0. I found it easiest to keep all the data in the same vm / lxc, pretty straight forward to maintain

[–] ikidd@lemmy.world 2 points 1 day ago

If you use Live Migrate, realize that it doesn't work on an LXC, only VMs. Your containers will be restarted with the LXC on the new node.

[–] just_another_person@lemmy.world 2 points 1 day ago (5 children)

Run Docker at the host level. Every level down from there is not only a knock to performance across the spectrum, it just makes a mess of networking. Anyone in here saying "it's easy to backup in a VM" has completely missed the point of containers, and apparently does not understand how to work with them.

You shouldn't ever need to backup containers, and if you're expecting data loss if one goes away, yerdewinitwrawng.

[–] Pika@sh.itjust.works 1 points 1 day ago* (last edited 1 day ago)

Just chiming in, this is not recommended for proxmox

The documentation (FAQ 13) actually directly says that docker should be installed as a QEMU VM on proxmox and that it should not be installed on the Proxmox VE Host

[–] Oisteink@feddit.nl 2 points 1 day ago (1 children)

You dont need or want docker on your vm host. But a bare metal docker host can solve many peoples needs.

load more comments (1 replies)
load more comments (3 replies)
[–] Dalraz@lemmy.ca 2 points 2 days ago

I personally like lxc's over vms for my home lab and i run a dedicated lxc for docker and one running a single node k8s.

[–] just_another_person@lemmy.world 1 points 1 day ago (4 children)

This thread has raised so many questions I'd like answered:

  1. Why are people backing up containers?
  2. Why are people running docker-in-docker?
  3. I saw someone mention snapshotting containers...what's the purpose of this?
  4. Why are people backing up docker installs?

Seriously thought I was going crazy reading some of these, and now I'm convinced the majority of people posting suggestions in here do not understand how to use containers at all.

Flat file configs, volumes, layers, versioning...it's like people don't know what these are are how to use them, and that is incredibly disconcerting.

[–] Oisteink@feddit.nl 4 points 1 day ago (1 children)

Because a lot of people don’t learn docker, they install docker because some software they want to use is distributed that way.

[–] mr_jaaay@lemmy.ml 2 points 1 day ago

Yup, this is me exactly. I’ve been planning on going more indepth but haven’t found the time. Inunderstand Linux and how to use LXCs, docker less so.

[–] DarkDarkHouse@lemmy.sdf.org 1 points 1 day ago* (last edited 1 day ago)
  1. I’m backing up LXCs, like I’d back up a VM. I don’t back up Docker containers, just their config and volumes.
  2. I don’t think anyone is doing that. We’re talking about installing Docker in LXC. One of the Proxmox rules you can live by is to not install software on the host. I don’t see the problem with installing Docker in an LXC for that reason.
  3. I’ll snapshot an LXC before running things like a dist-upgrade, or testing something that might break things. It’s very easy, so why not?
  4. I back up my LXC that has Docker installed because that way it’s easy to restore everything, including local volumes, to various points in time.
[–] mr_jaaay@lemmy.ml 2 points 1 day ago (1 children)

Follow-up question: do you have any good resources to start with for a simple overview on how we should be using containers? I’m not a developer, and from my experiences most documentation on the topic I’ve come across targets developers and devops people. As someone else mentioned, I use docker because it’s the way lots of things happen to be packaged - I’m more used to the Debian APT way of doing things.

[–] just_another_person@lemmy.world 2 points 1 day ago (1 children)

I don't have anything handy, but I see your point, and I'd shame lazy devs for not properly packaging things maybe 😂

You mentioned you use Proxmox, which is already an abstraction on bare-metal, so that's about as easy as easy an interface as I can imagine for a hosted machine without using something like Docker Desktop and using it to manage a machine remotely (not a good idea).

As a develop, I guess I was slightly confused on some suggestions on ways to use things being posted in this sub, but some of the responses I guess clarify that. There isn't enough simplicity in explaining the "what" of containers, so people just use them the simplest way they understand, which also happens to be the "wrong way". It's kind of hard to grasp that when you live with these things 24/7 for years. Kind of a similar deal with networking solutions like Tailscale where I see people installing it everywhere and not understanding why that's a bad idea 😂

So save you a lot of learning, I'll just not go down a rabbit hole if you just want something to work well. Ping back here if you get into a spot of trouble, and I'll definitely hop in to give a more detailed explanation on a workflow that is more effective than what it seems most people in here are using.

In fact, I may have just been inspired to do a write up on it.

[–] mr_jaaay@lemmy.ml 1 points 1 day ago (1 children)

Fair enough, would love to read something like this :-)

Yeah, I’ve been into Linux for 20 years, sometimes a bit on/off, as an all-around-sysadmin in mainly Windows places. And learned just enough of Docker to use it instead of apt - which I’d prefer, but as you said, many newer services don’t exist in debian repos or as .deb packages, only docker or similar.

[–] just_another_person@lemmy.world 1 points 1 day ago

If you're familiar with Linux, just read the Dockerfile of any given project. It's literally just a script for running a thing. You can take that info and install how you'd like if needed.

[–] sugar_in_your_tea@sh.itjust.works 1 points 1 day ago

I'm guessing people are largely using the wrong terminology for things that make more sense, like backing up/snapshotting config and data that containers use. Maybe they're also backing up images (which a lot of people call "containers"), just in case it gets yanked from wherever they got it from.

That said, yeah, someone should write a primer on how to use Docker properly and link it in the sidebar. Something like:

  1. docker-compose or podman for managing containers (a lot easier than docker run)
  2. how to use bind mounts and set permissions, as well as sharing volumes between containers (esp. useful if your TLS cert renewal is a separate container from your TLS server)
  3. docker networks - how to get containers to talk w/o exposing their ports system-wide (I only expose two ports, Caddy for TLS, and Jellyfin because my old smart TV can't seem to handle TLS)
  4. how tags work - i.e. when to use latest, the difference between <image>:<major>.<minor>.<patch> and <image>:<major>, etc, and updating images (i.e. what happens when you "pull")

I've been using docker for years, but I'm sure the are some best practices I am missing since I'm more of a developer than a sysadmin.

load more comments
view more: next ›