I’ve got a pile of hardware as well. 3 old sff desktops. A pile of 2.5gbe nics, ram, nvmes, and ssds to put in them. Proxmox iso on a thumb drive with Ventoy. All ready to go.
Gonna run kubernetes atop proxmox, and a bunch of stuff atop that.
Have another tower I set up for storage. I installed all the drives, and proxmox, and OMV…and just left it there.
Now I’m thinking I might put the drives in my tower, with the GPU, and giving it a small VM for OMV and a bigger VM for Moonlight. Then I can just have a nice silent NUC for my desktop. I have several of those ready to go as well.
Ive setup that proxmox cluster, down to the sff and and the 2.5gb nics. Ceph works decently at 2.5gb, but there are some foibles. Anything running sqlite will need to be on local disc instead of the ceph data store to sort out write errors. It wont be in HA by default, but you can still power it off and migrate it to the ceph store, and the move it if you need to do long term maintance.
I also have GPU passthrough setup on my jellyfin container, so that isnt in HA either. Still, worth the performance bump, even with an sff gpu.
Ooh good call on jellyfin I’ll have to keep that in mind. I hadn’t considered that jellyfin would probably want direct access to the igpu for QSV or any other re- encoding. All my nodes are specced the same, hardware wise…I wonder if I’ll be able to just pass down the igpu to the worker-node vms and have that available to all of them. I don’t need their displays, and all my SFFs have onboard serial in case I actually need to see what’s on console.
Maybe? I’m working with a hodgepodge of sffs, so each box is different. I wouldnt expect it to work with direct PCI passthrough even with your setup, but maybe at a different abstraction layer?
If you are going full hyperconverged cluster, look at implementing proxlb too for dynamic load balancing. Its my next project on the cluster.
I’ve got a pile of hardware as well. 3 old sff desktops. A pile of 2.5gbe nics, ram, nvmes, and ssds to put in them. Proxmox iso on a thumb drive with Ventoy. All ready to go.
Gonna run kubernetes atop proxmox, and a bunch of stuff atop that.
Have another tower I set up for storage. I installed all the drives, and proxmox, and OMV…and just left it there.
Now I’m thinking I might put the drives in my tower, with the GPU, and giving it a small VM for OMV and a bigger VM for Moonlight. Then I can just have a nice silent NUC for my desktop. I have several of those ready to go as well.
Ive setup that proxmox cluster, down to the sff and and the 2.5gb nics. Ceph works decently at 2.5gb, but there are some foibles. Anything running sqlite will need to be on local disc instead of the ceph data store to sort out write errors. It wont be in HA by default, but you can still power it off and migrate it to the ceph store, and the move it if you need to do long term maintance.
I also have GPU passthrough setup on my jellyfin container, so that isnt in HA either. Still, worth the performance bump, even with an sff gpu.
Ooh good call on jellyfin I’ll have to keep that in mind. I hadn’t considered that jellyfin would probably want direct access to the igpu for QSV or any other re- encoding. All my nodes are specced the same, hardware wise…I wonder if I’ll be able to just pass down the igpu to the worker-node vms and have that available to all of them. I don’t need their displays, and all my SFFs have onboard serial in case I actually need to see what’s on console.
Maybe? I’m working with a hodgepodge of sffs, so each box is different. I wouldnt expect it to work with direct PCI passthrough even with your setup, but maybe at a different abstraction layer?
If you are going full hyperconverged cluster, look at implementing proxlb too for dynamic load balancing. Its my next project on the cluster.