8700g with 256gb ram is possible on desktop. Half the APU performance, but less stupid bigger model > fast, for coding. No one seems to be using such a rig though.
To be fair: Motherboard, cpu and ram I bought 6 years ago with an nvidia 1660. Then I bought the Radeon RX 6600 XT on release in 2021, so 4 years ago. But it’s a generic gaming rig.
I would be surprised if 2000$ worth of modern hardware, picked for this specific task would be worse than that mini PC.
It will probably kick the ass of that desktop. $2000 won’t get you far with a conventional build.
8700g with 256gb ram is possible on desktop. Half the APU performance, but less stupid bigger model > fast, for coding. No one seems to be using such a rig though.
Well, thats what I said “AI optimized”.
Even my 5 year old 900$ rig can output like 4 tps.
With what model? GPT oss or something else?
LLama 3 8B Instruct: 25tps
DeepSeek R1 distill qwen 14b: 3.2tps
To be fair: Motherboard, cpu and ram I bought 6 years ago with an nvidia 1660. Then I bought the Radeon RX 6600 XT on release in 2021, so 4 years ago. But it’s a generic gaming rig.
I would be surprised if 2000$ worth of modern hardware, picked for this specific task would be worse than that mini PC.
I promise. It’s not possible. But things change quickly of course.
(Unless you’re lucky/pro and get your hands on some super cheap used high end hardware…)
To be honest that is pretty good. Thanks!
There is nothing “optimized” that will get you better inference performance of medium/large models at $2000.