suoko@feddit.it to LocalLLaMA@sh.itjust.worksEnglish · edit-28 days agoI dropped Qwen and Gemma for a 27B model that runs smoothly on my 8GB graphics card - Bonsai 27Bwww.xda-developers.comexternal-linkmessage-square20linkfedilinkarrow-up147arrow-down18
arrow-up139arrow-down1external-linkI dropped Qwen and Gemma for a 27B model that runs smoothly on my 8GB graphics card - Bonsai 27Bwww.xda-developers.comsuoko@feddit.it to LocalLLaMA@sh.itjust.worksEnglish · edit-28 days agomessage-square20linkfedilink
minus-squareBeefAndPoultry@lemmus.orglinkfedilinkEnglisharrow-up11·edit-210 days agoYeah RTX 2080 8GB and 32GB of system RAM, I wrote a guide https://lemmus.org/post/24235317 I’ll have to do an updated guide once Llama.cpp v0.7.0 comes out because there’s a new option now for MoE expert caching. It doesn’t work that well yet but I think they’ll continue to improve it. https://lemmus.org/post/25955310
Yeah RTX 2080 8GB and 32GB of system RAM, I wrote a guide
https://lemmus.org/post/24235317
I’ll have to do an updated guide once Llama.cpp v0.7.0 comes out because there’s a new option now for MoE expert caching. It doesn’t work that well yet but I think they’ll continue to improve it. https://lemmus.org/post/25955310
Thanks a lot 🙇🙇🙇🙇