I"m a few steps in. I have the hardware. I know what hypervisor I want to use, and I know some of the things I want it to run with plenty of fun sounding projects to get it to host things. Its nearly put together (just an extra cache disk to hook up with cables already ran for it so it’ll take 10 minutes to do). But it’s sat on my desk since March of this year. Unconfigured. Just there.
I’ve got a pile of hardware as well. 3 old sff desktops. A pile of 2.5gbe nics, ram, nvmes, and ssds to put in them. Proxmox iso on a thumb drive with Ventoy. All ready to go.
Gonna run kubernetes atop proxmox, and a bunch of stuff atop that.
Have another tower I set up for storage. I installed all the drives, and proxmox, and OMV…and just left it there.
Now I’m thinking I might put the drives in my tower, with the GPU, and giving it a small VM for OMV and a bigger VM for Moonlight. Then I can just have a nice silent NUC for my desktop. I have several of those ready to go as well.
Ive setup that proxmox cluster, down to the sff and and the 2.5gb nics. Ceph works decently at 2.5gb, but there are some foibles. Anything running sqlite will need to be on local disc instead of the ceph data store to sort out write errors. It wont be in HA by default, but you can still power it off and migrate it to the ceph store, and the move it if you need to do long term maintance.
I also have GPU passthrough setup on my jellyfin container, so that isnt in HA either. Still, worth the performance bump, even with an sff gpu.
Ooh good call on jellyfin I’ll have to keep that in mind. I hadn’t considered that jellyfin would probably want direct access to the igpu for QSV or any other re- encoding. All my nodes are specced the same, hardware wise…I wonder if I’ll be able to just pass down the igpu to the worker-node vms and have that available to all of them. I don’t need their displays, and all my SFFs have onboard serial in case I actually need to see what’s on console.
Maybe? I’m working with a hodgepodge of sffs, so each box is different. I wouldnt expect it to work with direct PCI passthrough even with your setup, but maybe at a different abstraction layer?
If you are going full hyperconverged cluster, look at implementing proxlb too for dynamic load balancing. Its my next project on the cluster.
But I’ve been having so much fun buying hardware for my homelab. I don’t want to actually build it and configure it, that’s too much like work.
I"m a few steps in. I have the hardware. I know what hypervisor I want to use, and I know some of the things I want it to run with plenty of fun sounding projects to get it to host things. Its nearly put together (just an extra cache disk to hook up with cables already ran for it so it’ll take 10 minutes to do). But it’s sat on my desk since March of this year. Unconfigured. Just there.
I’ve got a pile of hardware as well. 3 old sff desktops. A pile of 2.5gbe nics, ram, nvmes, and ssds to put in them. Proxmox iso on a thumb drive with Ventoy. All ready to go.
Gonna run kubernetes atop proxmox, and a bunch of stuff atop that.
Have another tower I set up for storage. I installed all the drives, and proxmox, and OMV…and just left it there.
Now I’m thinking I might put the drives in my tower, with the GPU, and giving it a small VM for OMV and a bigger VM for Moonlight. Then I can just have a nice silent NUC for my desktop. I have several of those ready to go as well.
Ive setup that proxmox cluster, down to the sff and and the 2.5gb nics. Ceph works decently at 2.5gb, but there are some foibles. Anything running sqlite will need to be on local disc instead of the ceph data store to sort out write errors. It wont be in HA by default, but you can still power it off and migrate it to the ceph store, and the move it if you need to do long term maintance.
I also have GPU passthrough setup on my jellyfin container, so that isnt in HA either. Still, worth the performance bump, even with an sff gpu.
Ooh good call on jellyfin I’ll have to keep that in mind. I hadn’t considered that jellyfin would probably want direct access to the igpu for QSV or any other re- encoding. All my nodes are specced the same, hardware wise…I wonder if I’ll be able to just pass down the igpu to the worker-node vms and have that available to all of them. I don’t need their displays, and all my SFFs have onboard serial in case I actually need to see what’s on console.
Maybe? I’m working with a hodgepodge of sffs, so each box is different. I wouldnt expect it to work with direct PCI passthrough even with your setup, but maybe at a different abstraction layer?
If you are going full hyperconverged cluster, look at implementing proxlb too for dynamic load balancing. Its my next project on the cluster.