tl;dr: Proxmox or bare metal? Containers yes/no? What is your use case?
I used a Dell R710 when I first started self-hosting, it ran ESXi with one VM for each service I wanted. Restarting the server required me to first shutdown each VM in order and then the whole host. When setting up a new service to host I had to create a new VM (allocate RAM, disk etc), install the OS (I ran Debian) and then follow instructions for how to setup the service. This mostly was not a problem but one software I never managed to get working was Apache Guacamole.
Nowadays I have a Dell Optiplex I salvaged for parts, got a new case and all my HDDs from my R710. Because it has much less RAM and an old Intel i5 (6th or 7th generation), I decided to get into Docker. With Docker containers you write your compose file and it will just work. No more need to dig through documentation for which version of a dependency to use, how to handle if two services on the same host need different versions (this was part of the reason for one VM/service). With Docker, I can try out a software in seconds and have it configured to my liking in minutes.
Today I have NixOS (bare metal) and it comes with Podman which uses systemd. Hence restarting my OS (albeit not that often, almost never unless I mess up my config) is a no-brainer because Podman via systemd will manage everything. Adding, stopping or removing containers in general is easy. I have a script running as a service which will stop a container, create a BTRFS subvolume snapshot, start the service again and start borg backup to backup from the snapshot.
For me using Proxmox would just an extra layer of complexity I don’t need. I only have one server and I am the only user.
Questions:
- Do you use Proxmox instead of a bare metal installation?
- Do you use containers or do you install manually?
- What is you use case that requires your setup the way it is?
I use both. The main benefit of proxmox is having VM snapshots/daily backups, if you mess up, just restore the whole thing. You also have stricter isolation, e.g. put public facing containers in one VM and local only stuff in another.
Also has high availability if you have multiple nodes but that can be achieved with container orchestration as well.
Couldnt you just make daily snapshots on baremetal? Like most configs will support a ZFS pool.
All my stuff is just configs anyway, so my backup is a Git commit. Lighter than a full snapshot. My server can be restarted with like 50mbs of information.
Docker images are images so disposable. All my files and databases are on my NAS which is a ZFS pool mirrored and with a weekly backup to a secondary NAS also mirrored. Plus I have an offsite cloud backup of everything. There isnt really anything on my server that matters other than some configs, scripts and ymls. Which by commiting to my Gitea, are all on my NAS with the rest of my data.
I still feel like Proxmox just invites you to tinker and fiddle with stuff that should be set and forget. Or overcomplexify your setup with too many moving parts.
For public stuff I just use a VPS. Better internet connection than at home, high availability and DDOS protection. Otherwise, an LXC with good firewall rules and a well configured reverse proxy should be sufficient.
Sure, you can probably build something similar with zfs snapshots, but it’ll likely not be as convenient. Have a VM? Create daily snapshots. Break something and can’t figure it out? Just reset to the last backup to a complete working state.
Personally, I have the containers’ mounts in the VMs (ergo on the nodes’ disks) and just using the NAS for the backups. Even if both Proxmox nodes catch fire I can just import the snapshot to a new one and have a working VM in the same state again. That’s a whole different level from re-cloning all the compose stacks, mount NFS shares etc. It’s the ease of use.