This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
dwa3592
Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally.
If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider.
I estimate that the server consumes probably around 500W during inference.
In Germany where 1kwh cost around 0.3USD, 18k tokens inferred locally would therefore cost 0.15USD which is 30x the costs of using an inference provider.
But for ppl who worry about their data, running locally might still be good. However, they should be aware, that it is much less efficient than using an inference provider.
The efficiency gap will also significantly increase as new GPUs will make inference much more efficient.
EDIT: I first thought it'd be 180k token, but thanks to someone mentioning in the comments, it is 18k. I guess with that, it will be tough unless u got electricity almost for free. Also, the inference providers are probably still using H200/H100 for those small models. Once they use GB300 or next year the new Ruby GPUs, inference will be cheaper by a factor of 30. By then, running local models will mostly be about privacy.
hagen8
Here's my report running several different models on a dual Xeon with 256 GB of DDR4 and no GPU.
The 5.2 tokens per second generation is not that bad, what kills it is the 16.2 prompt processing that makes this too slow to consider even if you have the hardware lying around.
A dual Xeon of this era is probably pulling 300W or more when loaded.
At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space.
If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day.
That’s about $0.30 per million output tokens.
Coincidentally, that’s the same price for this model on OpenRouter right now, but OpenRouter token gen will be 8X faster.
There are a lot of good reasons to experiment with running LLMs locally, like if you don’t want any data leaving your house.
Don’t think that you’re going to come out ahead monetarily. I say this as someone with a lot more money invested in local inference hardware at home. It’s fun, but it’s not a way to save money.
Aurornis
He's shown me his set up in his basement. It's sick! Talk about your 3d printer next!
deltamidway
I love my little dual core X99 board with Xeon E5 2673 V3. It's not power efficient, but I just leave it in my basement for local Jupyter Notebook stuff. Much faster than everything cloud-based for a reasonably price at my scale.
rhema
I run the same setup Gemma 4 26B on a 2013 Mac Pro (dual graphics cards but they're useless for this). I also get about 5 t/s. It's perfectly serviceable for some tasks!
comments (10)
I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat
This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
dwa3592
If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider.
I estimate that the server consumes probably around 500W during inference.
In Germany where 1kwh cost around 0.3USD, 18k tokens inferred locally would therefore cost 0.15USD which is 30x the costs of using an inference provider.
But for ppl who worry about their data, running locally might still be good. However, they should be aware, that it is much less efficient than using an inference provider.
The efficiency gap will also significantly increase as new GPUs will make inference much more efficient.
EDIT: I first thought it'd be 180k token, but thanks to someone mentioning in the comments, it is 18k. I guess with that, it will be tough unless u got electricity almost for free. Also, the inference providers are probably still using H200/H100 for those small models. Once they use GB300 or next year the new Ruby GPUs, inference will be cheaper by a factor of 30. By then, running local models will mostly be about privacy.
hagen8
https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5...
hparadiz
https://news.ycombinator.com/item?id=48354801
throwaway2027
throwawayffffas
neomindryan
At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space.
If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day.
That’s about $0.30 per million output tokens.
Coincidentally, that’s the same price for this model on OpenRouter right now, but OpenRouter token gen will be 8X faster.
There are a lot of good reasons to experiment with running LLMs locally, like if you don’t want any data leaving your house.
Don’t think that you’re going to come out ahead monetarily. I say this as someone with a lot more money invested in local inference hardware at home. It’s fun, but it’s not a way to save money.
Aurornis
deltamidway
rhema
broabprobe