Release v0.7.0 · ggml-org/llama.cpp
github.com
Overview llama.cpp v0.7.0 adds the LiliCorr speculative decoding technique and MTP for GLM5-Next, a new GPU cache for MoE experts kept in host memory (--moe-cache-mib, up to ~2.2x decode speedup wh...

--moe-cache-mib doesn’t seem optimized for tight VRAM situations or if your PCIe link gets saturated by it, but it improves performance on some setups.

  • trewq@lemmy.worldEnglish
    3·
    6 days ago

    Model manager already exist for quite a while, you can load and unload models via webui. I personally uses preset file option. This is new version of the existing mgr

    • 9tr6gyp3@lemmy.worldEnglish
      1·
      6 days ago

      Ah, i must have missed it. Thanks for the info 🙂

      Hmm, not seeing it directly in the llama.cpp 0.6.0 version. Where were you seeing it?

      • trewq@lemmy.worldEnglish
        2·
        6 days ago

        Should be added sometime early this year. I think it was mentioned in HF blog/article when i was looking for way to use multiple models without restarting llama