This one is a Mixture of Experts model (40B total weight, 2.5B active during token generation). Inference engines like llama.cpp have flags you can pass to offload the lesser used experts to system RAM.
It’s slower than having the entire model resident in VRAM, but generally pretty usable.
This one is a Mixture of Experts model (40B total weight, 2.5B active during token generation). Inference engines like llama.cpp have flags you can pass to offload the lesser used experts to system RAM.
It’s slower than having the entire model resident in VRAM, but generally pretty usable.