You’re probably burning more energy turning it off and on again. It doesn’t really use any noticeable power sitting idle.
Anyway, a direct comparison would be pretty difficult because your model is probably tens of billions of parameters, not over a trillion. Energy consumption per output token will probably be a bit higher for the frontier models but something that people have found is that higher quality models often need fewer tokens to achieve the same goal. Plus how many times do you re-prompt your local model vs Claude Fable or Opus for example to get the desired result?
You’re probably burning more energy turning it off and on again. It doesn’t really use any noticeable power sitting idle.
I am absolutely not burning more energy than a frontier model by doing things like putting my laptop to sleep or shutting down unused services when I want to conserve battery power.
Anyway, a direct comparison would be pretty difficult because your model is probably tens of billions of parameters, not over a trillion.
True.
Energy consumption per output token will probably be a bit higher for the frontier models but something that people have found is that higher quality models often need fewer tokens to achieve the same goal.
That’s actually not true. In fact it’s much the opposite. Frontier models churn through tokens at a much higher rate, because of their higher complexity and higher number of parameters. Research is still new on this, but having a frontier model analyze your code files versus a small, local model for the same task seems to be enormously wasteful. If you must use a frontier model for something, have it do that work after receiving the output from an agent using a small model to read and summarize your code.
Plus how many times do you re-prompt your local model vs Claude Fable or Opus for example to get the desired result?
…Almost never? I’m not a fan of letting AI do much of ANY of my coding, because it will inevitably bloat my codebase with garbage regardless of which model I use. So I severely restrict my model usage to simple, clearly-defined, narrow-scoped tasks that can save me a bit of time, and that’s it. With guardrails and discipline like that, I barely ever have the need to re-prompt.
You’re probably burning more energy turning it off and on again. It doesn’t really use any noticeable power sitting idle.
Anyway, a direct comparison would be pretty difficult because your model is probably tens of billions of parameters, not over a trillion. Energy consumption per output token will probably be a bit higher for the frontier models but something that people have found is that higher quality models often need fewer tokens to achieve the same goal. Plus how many times do you re-prompt your local model vs Claude Fable or Opus for example to get the desired result?
I am absolutely not burning more energy than a frontier model by doing things like putting my laptop to sleep or shutting down unused services when I want to conserve battery power.
True.
That’s actually not true. In fact it’s much the opposite. Frontier models churn through tokens at a much higher rate, because of their higher complexity and higher number of parameters. Research is still new on this, but having a frontier model analyze your code files versus a small, local model for the same task seems to be enormously wasteful. If you must use a frontier model for something, have it do that work after receiving the output from an agent using a small model to read and summarize your code.
…Almost never? I’m not a fan of letting AI do much of ANY of my coding, because it will inevitably bloat my codebase with garbage regardless of which model I use. So I severely restrict my model usage to simple, clearly-defined, narrow-scoped tasks that can save me a bit of time, and that’s it. With guardrails and discipline like that, I barely ever have the need to re-prompt.