BeefAndPoultry@lemmus.org to LocalLLaMA@sh.itjust.worksEnglish · 5 days agoRecursiveSI/flint 40b-a2.5bhuggingface.coexternal-linkmessage-square4linkfedilinkarrow-up115arrow-down11file-text
arrow-up114arrow-down1external-linkRecursiveSI/flint 40b-a2.5bhuggingface.coBeefAndPoultry@lemmus.org to LocalLLaMA@sh.itjust.worksEnglish · 5 days agomessage-square4linkfedilinkfile-text
minus-squareBeefAndPoultry@lemmus.orgOPlinkfedilinkEnglisharrow-up2·5 days agoIt’s a MoE model so it’s much more flexible than a dense model, see my guide for llama.cpp https://lemmus.org/post/24235317 Now there’s also --moe-cache-mib that could beat --n-cpu-moe depending on your exact hardware. I’ll need to write an updated guide. And now there’s Strata and PolyStrata which get even better performance than that.
It’s a MoE model so it’s much more flexible than a dense model, see my guide for llama.cpp https://lemmus.org/post/24235317
Now there’s also
--moe-cache-mibthat could beat--n-cpu-moedepending on your exact hardware. I’ll need to write an updated guide.And now there’s Strata and PolyStrata which get even better performance than that.