Sure, but adding adversarial loops, or, some complex addition to every prompt would increase the token usage massively, thus the cost.
So, why bother with that?
I could very well be wrong, but all indications I’ve seen have been that that the cost of the tokens needed to get something even vaguely acceptable is greater than the cost of just hiring someone, or a group, to do the job. Unless something needs to be done fast due to an immediate concern, and the cost is no object.
Well yes, I’m not saying it’s a free lunch. The point is a large up-front investment buys you unlimited tokens in perpetuity for no more than the cost of electricity to power the cluster — which can be surprisingly cheap for UMA machines.
I’ve swapped off API models entirely and have been able to do a lot more useful work since I don’t have to worry about token cost or usage limits. Granted it’s slower per task (less tk/s), but overall much more productive, and private.
Sure, but adding adversarial loops, or, some complex addition to every prompt would increase the token usage massively, thus the cost.
So, why bother with that?
I could very well be wrong, but all indications I’ve seen have been that that the cost of the tokens needed to get something even vaguely acceptable is greater than the cost of just hiring someone, or a group, to do the job. Unless something needs to be done fast due to an immediate concern, and the cost is no object.
Perhaps they run a local model, then tokens are essentially free.
Would need a very beefy cluster, and that definitely isn’t free.
Well yes, I’m not saying it’s a free lunch. The point is a large up-front investment buys you unlimited tokens in perpetuity for no more than the cost of electricity to power the cluster — which can be surprisingly cheap for UMA machines.
I’ve swapped off API models entirely and have been able to do a lot more useful work since I don’t have to worry about token cost or usage limits. Granted it’s slower per task (less tk/s), but overall much more productive, and private.