If you’re considering video transcoding, I’d give Intel a look. Quicksync is pretty well supported across all of the media platforms. I do think Jellyfin is on a much more modern ffmpeg than Plex, and it actually supports AMD. But, I don’t have any experience with that… Only Nvidia and Intel. You really don’t need a powerful CPU either. I’ve got my Plex server on a little i5 NUC, and it can do 4k transcodes no problem.
You really don’t need an AIO with a 5600X. Just grab a reasonably sized tower cooler and call it a day. There’s less to fail, and less risk of water damage if it fails catastrophically. I’ve found thermalright to be exceptionally good for how well priced they are. Not as quiet as Noctua, but damn near the same cooling performance.
Another thing to consider is that a 5600X doesn’t have built in graphics. I think you’d need to jump up to AM5/7600X for that.
I’d take a look at packer and ansible. Packer can be used to prepare a new base image for your VMs. And ansible can be used to automate the provisioning of a VM once it’s booted.