--moe-cache-mib doesn’t seem optimized for tight VRAM situations or if your PCIe link gets saturated by it, but it improves performance on some setups.
--moe-cache-mib doesn’t seem optimized for tight VRAM situations or if your PCIe link gets saturated by it, but it improves performance on some setups.
Im actually more excited about the model manager here. To be able to load/unload/switch models easily is something I had scripts do, but now its built-in.
Model manager already exist for quite a while, you can load and unload models via webui. I personally uses preset file option. This is new version of the existing mgr
Ah, i must have missed it. Thanks for the info 🙂
Hmm, not seeing it directly in the llama.cpp 0.6.0 version. Where were you seeing it?
Should be added sometime early this year. I think it was mentioned in HF blog/article when i was looking for way to use multiple models without restarting llama
Just use llama-swap