--moe-cache-mib doesn’t seem optimized for tight VRAM situations or if your PCIe link gets saturated by it, but it improves performance on some setups.
--moe-cache-mib doesn’t seem optimized for tight VRAM situations or if your PCIe link gets saturated by it, but it improves performance on some setups.
FWIW, vLLM already had a lot of this. This makes it so you can run decent models on smaller devices like phones or Raspberry Pis.
Getting pretty good results with llama.cpp and Bonsai on laptops and older phones.