NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.
This might actually work in 8GB VRAM? 74B parameters at 24GB means about 3GB for the 9b active parameters.
Would’ve been nice if the model was made to fit in 22GB VRAM, so one could fit it together with a graphical desktop environment on a 24GB GPU… But I guess we’re not NVidia’s intended user base.
Well it’s a MoE, so at least in llama.cpp you can offload some experts and use the new MoE caching
--moe-cache-miband you’ll probably still get at least 90% of the speed